2023-09-30T00:48:01.067 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:48:01.068 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806 -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/orig.config.yaml 2023-09-30T00:48:01.105 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:48:01.189 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} email: yweinste@redhat.com first_in_suite: false job_id: '7406806' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk0GqMes8Qq3IX+gKj0dc7tzXhqKRmxhbppzXWrwEVrK/bWaCoqihwFWCgphmoKWY4NckYAsXjhGGM2ActeLA8= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFLTwF6gV082yTB3YMXebeTq3Z8SDVUY/zK81SDcvj45/0aCs9JXS8H7mD7/i7AtGoa7K0/sIYaPN1IHe50es0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:48:01.190 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:48:01.190 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:48:01.190 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:48:01.192 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:48:01.214 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:48:01.215 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:48:01.215 INFO:teuthology.packaging:ref: None 2023-09-30T00:48:01.215 INFO:teuthology.packaging:tag: None 2023-09-30T00:48:01.215 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:48:01.215 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:48:01.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:48:01.367 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:48:01.367 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:48:01.373 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:48:01.373 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:48:01.403 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:48:01.417 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:48:01.437 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:48:01.463 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:42:28.713986', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk0GqMes8Qq3IX+gKj0dc7tzXhqKRmxhbppzXWrwEVrK/bWaCoqihwFWCgphmoKWY4NckYAsXjhGGM2ActeLA8='} 2023-09-30T00:48:01.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:42:28.712583', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFLTwF6gV082yTB3YMXebeTq3Z8SDVUY/zK81SDcvj45/0aCs9JXS8H7mD7/i7AtGoa7K0/sIYaPN1IHe50es0='} 2023-09-30T00:48:01.488 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:48:01.494 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T00:48:01.494 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T00:48:01.495 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:48:01.603 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda0ff61ee0>, signals=[15]) 2023-09-30T00:48:01.604 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:48:01.610 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:48:01.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T00:48:01.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:48:01.678 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:48:01.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:48:01.746 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:48:01.753 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2023-09-30T00:48:01.776 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2023-09-30T00:48:01.836 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2023-09-30T00:48:01.836 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:48:01.837 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:48:01.838 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-09-30T00:48:01.863 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2023-09-30T00:48:01.886 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2023-09-30T00:48:01.886 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2023-09-30T00:48:01.948 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2023-09-30T00:48:01.948 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8" 2023-09-30T00:48:01.948 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2023-09-30T00:48:01.948 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:48:01.948 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:48:01.949 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:48:01.950 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-09-30T00:48:01.971 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:48:01.980 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:48:02.004 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:48:02.004 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:48:02.007 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:48:02.027 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:48:02.037 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:48:02.037 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:48:02.067 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:48:02.107 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:48:02.161 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:48:02.181 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:48:02.181 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:48:02.181 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:48:02.182 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:48:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:48:02.201 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T00:48:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:48:02.203 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-09-30T00:48:02.261 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:48:02.261 INFO:teuthology.task.kernel:Running kernel on smithi157: 4.18.0-489.el8.x86_64 2023-09-30T00:48:02.261 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2023-09-30T00:48:02.263 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:48:02.263 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-489.el8.x86_64 2023-09-30T00:48:02.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2023-09-30T00:48:06.228 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 24 MB/s | 33 MB 00:01 2023-09-30T00:48:07.091 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-09-30T00:48:13.986 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 48 MB 00:01 2023-09-30T00:48:14.416 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 36 MB/s | 48 MB 00:01 2023-09-30T00:48:21.095 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 72 kB/s | 18 kB 00:00 2023-09-30T00:48:21.657 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-09-30T00:48:21.812 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T00:48:22.088 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-09-30T00:48:22.232 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:48:22.517 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-30T00:48:23.991 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2023-09-30T00:48:24.387 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 16 MB 00:01 2023-09-30T00:48:27.798 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 414 kB/s | 24 kB 00:00 2023-09-30T00:48:28.144 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 560 kB/s | 24 kB 00:00 2023-09-30T00:48:31.305 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:48:31.305 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:48:31.576 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:48:31.576 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:48:31.578 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T00:48:31.579 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:48:31.579 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout:Install 3 Packages 2023-09-30T00:48:31.580 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:48:31.581 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 88 M 2023-09-30T00:48:31.581 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 96 M 2023-09-30T00:48:31.581 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T00:48:31.819 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:48:31.820 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout:Install 3 Packages 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 88 M 2023-09-30T00:48:31.821 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 96 M 2023-09-30T00:48:31.822 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T00:48:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-09-30T00:48:33.115 INFO:teuthology.orchestra.run.smithi157.stdout:(1/3): kernel-core-4.18.0-514.el8.x86_64.rpm 31 MB/s | 43 MB 00:01 2023-09-30T00:48:33.254 INFO:teuthology.orchestra.run.smithi115.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 25 MB/s | 35 MB 00:01 2023-09-30T00:48:33.682 INFO:teuthology.orchestra.run.smithi157.stdout:(2/3): kernel-4.18.0-514.el8.x86_64.rpm 5.2 MB/s | 10 MB 00:01 2023-09-30T00:48:33.698 INFO:teuthology.orchestra.run.smithi115.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 23 MB/s | 43 MB 00:01 2023-09-30T00:48:33.698 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:48:33.698 INFO:teuthology.orchestra.run.smithi115.stdout:Total 47 MB/s | 88 MB 00:01 2023-09-30T00:48:34.316 INFO:teuthology.orchestra.run.smithi157.stdout:(3/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 14 MB/s | 35 MB 00:02 2023-09-30T00:48:34.316 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:48:34.316 INFO:teuthology.orchestra.run.smithi157.stdout:Total 32 MB/s | 88 MB 00:02 2023-09-30T00:48:34.372 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T00:48:34.801 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T00:48:34.801 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T00:48:34.989 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T00:48:35.407 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T00:48:35.407 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T00:48:36.021 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T00:48:36.040 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T00:48:36.623 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T00:48:36.636 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T00:48:38.787 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T00:48:39.392 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:48:39.413 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T00:48:39.954 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:48:42.935 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:48:43.341 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:48:43.498 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:48:43.903 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:48:46.718 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:48:47.012 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:48:47.287 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:48:47.591 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:08.739 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:08.765 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.106 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.106 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:49:09.106 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:49:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.452 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:49:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:49:09.817 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:09.817 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:49:09.817 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T00:49:09.818 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:49:09.818 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:49:09.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:49:09.818 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T00:49:09.968 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:49:10.182 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:49:10.182 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.183 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.183 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:49:10.183 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:49:10.183 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:49:10.183 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:49:10.183 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:49:10.392 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:49:10.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.02 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 129.el8 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Group : System Environment/Base 2023-09-30T00:49:10.426 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 9265586 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Relocations : (not relocatable) 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : CentOS BuildSystem 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2023-09-30T00:49:10.427 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:49:10.428 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:49:10.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:49:10.429 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:49:10.479 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T00:49:10.615 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:49:10.874 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:49:10.874 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.875 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:49:10.875 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:49:10.875 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:49:10.875 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T00:49:10.875 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:49:10.875 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:49:11.083 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:49:11.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:49:11.084 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2023-09-30T00:49:11.117 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2023-09-30T00:49:11.117 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2023-09-30T00:49:11.117 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.02 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 129.el8 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Group : System Environment/Base 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 9265586 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:49:11.118 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:Relocations : (not relocatable) 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : CentOS BuildSystem 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:49:11.119 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2023-09-30T00:49:11.120 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:49:11.120 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:49:11.120 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:49:11.120 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:49:11.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:49:12.193 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2023-09-30T00:49:12.815 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2023-09-30T00:49:14.368 INFO:teuthology.orchestra.run.smithi115.stderr:done 2023-09-30T00:49:14.369 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T00:49:14.411 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.LIpypScRSm 2023-09-30T00:49:14.412 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LIpypScRSm 2023-09-30T00:49:14.485 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.LIpypScRSm 2023-09-30T00:49:14.626 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.LIpypScRSm is 6KB 2023-09-30T00:49:14.674 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.LIpypScRSm 2023-09-30T00:49:14.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:49:14.764 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:49:14.765 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:49:14.765 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:49:14.765 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:49:14.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:49:14.954 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2023-09-30T00:49:15.466 INFO:teuthology.orchestra.run.smithi157.stderr:done 2023-09-30T00:49:15.468 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T00:49:15.507 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.gzy7qRV45X 2023-09-30T00:49:15.507 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gzy7qRV45X 2023-09-30T00:49:15.581 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.gzy7qRV45X 2023-09-30T00:49:15.734 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.gzy7qRV45X is 6KB 2023-09-30T00:49:15.782 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.gzy7qRV45X 2023-09-30T00:49:15.800 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:49:15.870 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:49:15.870 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:49:15.870 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:49:15.870 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:49:15.871 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:49:16.034 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2023-09-30T00:49:44.982 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:49:44.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:49:44.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:49:46.038 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T00:49:46.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:49:46.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:03.515 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-09-30T00:50:04.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:50:12.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:50:12.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:13.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:50:13.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:15.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-09-30T00:50:16.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:50:27.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:50:27.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:28.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:50:28.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:31.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:50:46.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-09-30T00:50:46.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:50:46.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:51:01.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:51:01.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:51:01.344 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-09-30T00:51:01.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:51:01.650 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:51:01.650 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-09-30T00:51:01.667 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:51:01.667 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:51:01.667 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:51:01.667 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:51:02.668 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:51:02.669 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:51:02.751 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2023-09-30T00:51:02.765 DEBUG:teuthology.parallel:result is None 2023-09-30T00:51:05.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:51:23.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:51:23.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:51:23.610 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T00:51:23.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:51:23.905 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:51:23.905 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T00:51:23.924 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:51:23.924 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:51:23.924 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:51:23.924 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:51:24.925 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T00:51:24.925 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:51:24.994 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2023-09-30T00:51:25.010 DEBUG:teuthology.parallel:result is None 2023-09-30T00:51:25.010 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:51:25.019 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:51:25.019 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:51:25.023 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:51:25.044 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:51:25.058 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:51:25.109 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:51:25.109 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:51:25.112 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:51:25.152 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:51:25.158 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:51:25.158 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:51:25.189 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:51:25.227 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:51:25.239 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:51:25.253 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:51:25.265 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:51:25.267 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:51:25.274 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:51:25.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:51:25.276 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:51:25.311 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:51:25.362 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:51:25.363 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:51:25.365 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:51:25.385 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:51:25.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:51:25.530 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:51:25.576 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:51:25.619 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T00:51:25.619 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:51:25.686 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:51:25.733 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:51:25.793 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:51:25.880 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:51:25.925 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:51:25.925 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:51:25.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2023-09-30T00:51:25.995 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2023-09-30T00:51:26.067 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:51:26.090 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:51:26.528 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:51:26.538 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:51:26.538 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:51:26.598 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:51:26.669 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:51:26.670 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2023-09-30T00:51:26.760 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2023-09-30T00:51:26.761 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2023-09-30T00:51:26.833 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2023-09-30T00:51:26.835 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:51:26.835 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2023-09-30T00:51:26.888 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2023-09-30T00:51:26.888 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2023-09-30T00:51:26.912 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2023-09-30T00:51:26.912 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:51:26.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:51:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:51:26.941 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:51:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:51:26.981 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:51:26.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:51:27.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:51:27.048 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:51:27.058 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:51:27.058 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:51:27.075 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:51:27.076 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:58:57.786 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2023-09-30T00:58:57.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:58:57.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:58:57.856 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-09-30T00:58:57.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T00:58:57.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:58:57.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:58:57.995 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T00:58:58.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:58:58.070 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:58:58.083 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:58:58.084 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:58:58.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:58:58.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:58:58.088 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:58:58.116 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:58:58.130 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:58:58.144 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:58:58.158 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:58:58.200 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2023-09-30T00:58:58.212 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2023-09-30T00:58:58.226 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:58:58.228 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2023-09-30T00:58:58.239 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:58:58.240 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2023-09-30T00:58:58.253 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:58:58.267 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:58:58.306 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-09-30T00:58:58.310 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:58:58.310 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-09-30T00:58:58.310 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.310 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.310 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.311 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.331 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-09-30T00:58:58.335 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:58:58.335 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-09-30T00:58:58.336 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.336 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.336 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.336 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:58:58.337 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:58:58.347 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:58:58.347 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:58:58.347 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:58:58.347 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:58:58.353 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:58:58.353 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:58:58.354 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:58:58.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:58:58.356 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:58:58.356 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:58:58.502 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:58:58.502 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:58:58.507 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:58:58.508 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:58:58.668 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:58:58.668 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T00:58:58.668 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:58:58.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:58:58.675 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:58:58.675 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:58:58.697 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:58:58.698 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:58:58.704 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:58:58.704 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:58:58.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:58:58.773 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:58:58.843 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:58:58.852 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:58:58.909 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2023-09-30T00:58:58.911 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2023-09-30T00:58:58.919 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2023-09-30T00:58:58.920 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2023-09-30T00:58:59.182 INFO:teuthology.orchestra.run.smithi115.stdout:76 files removed 2023-09-30T00:58:59.195 INFO:teuthology.orchestra.run.smithi157.stdout:76 files removed 2023-09-30T00:58:59.210 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2023-09-30T00:58:59.223 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2023-09-30T00:58:59.979 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 215 kB/s | 78 kB 00:00 2023-09-30T00:58:59.993 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 215 kB/s | 78 kB 00:00 2023-09-30T00:59:00.360 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2023-09-30T00:59:00.363 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 26 kB/s | 9.2 kB 00:00 2023-09-30T00:59:00.652 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2023-09-30T00:59:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2023-09-30T00:59:02.261 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 21 MB/s | 33 MB 00:01 2023-09-30T00:59:02.306 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 21 MB/s | 33 MB 00:01 2023-09-30T00:59:12.244 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 48 MB 00:03 2023-09-30T00:59:12.513 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 48 MB 00:04 2023-09-30T00:59:19.444 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 71 kB/s | 18 kB 00:00 2023-09-30T00:59:19.832 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-09-30T00:59:20.012 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-09-30T00:59:20.423 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:59:22.180 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.5 MB 00:02 2023-09-30T00:59:22.590 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.5 MB 00:02 2023-09-30T00:59:23.623 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-30T00:59:23.975 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-09-30T00:59:25.619 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-09-30T00:59:27.276 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2023-09-30T00:59:29.584 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 564 kB/s | 24 kB 00:00 2023-09-30T00:59:31.198 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 335 kB/s | 24 kB 00:00 2023-09-30T00:59:33.961 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T00:59:33.964 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:59:33.964 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T00:59:33.964 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:59:33.964 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T00:59:33.964 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:59:33.965 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:59:33.966 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout:Install 21 Packages 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2023-09-30T00:59:33.967 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T00:59:33.968 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 63 M 2023-09-30T00:59:33.968 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T00:59:35.009 INFO:teuthology.orchestra.run.smithi115.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 8.1 MB/s | 5.2 MB 00:00 2023-09-30T00:59:35.059 INFO:teuthology.orchestra.run.smithi115.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 490 kB/s | 24 kB 00:00 2023-09-30T00:59:35.193 INFO:teuthology.orchestra.run.smithi115.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 5.8 MB/s | 796 kB 00:00 2023-09-30T00:59:35.285 INFO:teuthology.orchestra.run.smithi115.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 5.4 MB/s | 506 kB 00:00 2023-09-30T00:59:35.580 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T00:59:35.582 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:59:35.583 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:59:35.584 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T00:59:35.585 INFO:teuthology.orchestra.run.smithi157.stdout:Install 21 Packages 2023-09-30T00:59:35.586 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2023-09-30T00:59:35.586 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:59:35.586 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 63 M 2023-09-30T00:59:35.586 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T00:59:35.710 INFO:teuthology.orchestra.run.smithi115.stdout:(5/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 13 MB/s | 17 MB 00:01 2023-09-30T00:59:35.777 INFO:teuthology.orchestra.run.smithi115.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 671 kB/s | 45 kB 00:00 2023-09-30T00:59:35.861 INFO:teuthology.orchestra.run.smithi115.stdout:(7/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 9.7 MB/s | 5.6 MB 00:00 2023-09-30T00:59:35.913 INFO:teuthology.orchestra.run.smithi115.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 851 kB/s | 115 kB 00:00 2023-09-30T00:59:35.939 INFO:teuthology.orchestra.run.smithi115.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.4 MB/s | 190 kB 00:00 2023-09-30T00:59:35.981 INFO:teuthology.orchestra.run.smithi115.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 5.4 MB/s | 370 kB 00:00 2023-09-30T00:59:36.006 INFO:teuthology.orchestra.run.smithi115.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 5.2 MB/s | 358 kB 00:00 2023-09-30T00:59:36.031 INFO:teuthology.orchestra.run.smithi115.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.1 MB/s | 107 kB 00:00 2023-09-30T00:59:36.240 INFO:teuthology.orchestra.run.smithi115.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-09-30T00:59:36.265 INFO:teuthology.orchestra.run.smithi115.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2023-09-30T00:59:36.291 INFO:teuthology.orchestra.run.smithi115.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 875 kB/s | 44 kB 00:00 2023-09-30T00:59:36.416 INFO:teuthology.orchestra.run.smithi115.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-09-30T00:59:36.824 INFO:teuthology.orchestra.run.smithi115.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 144 kB/s | 59 kB 00:00 2023-09-30T00:59:36.891 INFO:teuthology.orchestra.run.smithi115.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2023-09-30T00:59:36.975 INFO:teuthology.orchestra.run.smithi115.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 500 kB/s | 75 kB 00:00 2023-09-30T00:59:37.250 INFO:teuthology.orchestra.run.smithi115.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2023-09-30T00:59:37.649 INFO:teuthology.orchestra.run.smithi157.stdout:(1/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 9.5 MB/s | 17 MB 00:01 2023-09-30T00:59:37.691 INFO:teuthology.orchestra.run.smithi157.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 586 kB/s | 24 kB 00:00 2023-09-30T00:59:37.800 INFO:teuthology.orchestra.run.smithi157.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 7.2 MB/s | 796 kB 00:00 2023-09-30T00:59:37.834 INFO:teuthology.orchestra.run.smithi115.stdout:(21/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 6.8 MB/s | 23 MB 00:03 2023-09-30T00:59:37.875 INFO:teuthology.orchestra.run.smithi157.stdout:(4/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 2.6 MB/s | 5.2 MB 00:01 2023-09-30T00:59:37.892 INFO:teuthology.orchestra.run.smithi115.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.8 MB/s | 3.5 MB 00:00 2023-09-30T00:59:37.917 INFO:teuthology.orchestra.run.smithi157.stdout:(5/23): libradosstriper1-17.2.6-1165.g93dce5e1. 4.2 MB/s | 506 kB 00:00 2023-09-30T00:59:37.967 INFO:teuthology.orchestra.run.smithi157.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 894 kB/s | 45 kB 00:00 2023-09-30T00:59:38.017 INFO:teuthology.orchestra.run.smithi157.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 2.3 MB/s | 115 kB 00:00 2023-09-30T00:59:38.084 INFO:teuthology.orchestra.run.smithi157.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.8 MB/s | 190 kB 00:00 2023-09-30T00:59:38.160 INFO:teuthology.orchestra.run.smithi157.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 4.8 MB/s | 370 kB 00:00 2023-09-30T00:59:38.235 INFO:teuthology.orchestra.run.smithi157.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 4.7 MB/s | 358 kB 00:00 2023-09-30T00:59:38.294 INFO:teuthology.orchestra.run.smithi157.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.8 MB/s | 107 kB 00:00 2023-09-30T00:59:38.519 INFO:teuthology.orchestra.run.smithi157.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-09-30T00:59:38.578 INFO:teuthology.orchestra.run.smithi157.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-09-30T00:59:38.662 INFO:teuthology.orchestra.run.smithi157.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 527 kB/s | 44 kB 00:00 2023-09-30T00:59:38.878 INFO:teuthology.orchestra.run.smithi157.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 217 kB/s | 47 kB 00:00 2023-09-30T00:59:39.270 INFO:teuthology.orchestra.run.smithi157.stdout:(16/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 4.0 MB/s | 5.6 MB 00:01 2023-09-30T00:59:39.296 INFO:teuthology.orchestra.run.smithi157.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 734 kB/s | 306 kB 00:00 2023-09-30T00:59:39.379 INFO:teuthology.orchestra.run.smithi157.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 902 kB/s | 75 kB 00:00 2023-09-30T00:59:39.468 INFO:teuthology.orchestra.run.smithi115.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 1.6 MB/s | 3.6 MB 00:02 2023-09-30T00:59:39.471 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:59:39.472 INFO:teuthology.orchestra.run.smithi115.stdout:Total 11 MB/s | 63 MB 00:05 2023-09-30T00:59:39.513 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T00:59:39.538 INFO:teuthology.orchestra.run.smithi157.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 220 kB/s | 59 kB 00:00 2023-09-30T00:59:39.559 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T00:59:39.559 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T00:59:39.605 INFO:teuthology.orchestra.run.smithi157.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 7.7 MB/s | 1.7 MB 00:00 2023-09-30T00:59:39.929 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T00:59:39.932 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T00:59:39.989 INFO:teuthology.orchestra.run.smithi157.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 9.4 MB/s | 3.6 MB 00:00 2023-09-30T00:59:40.230 INFO:teuthology.orchestra.run.smithi157.stdout:(22/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 5.4 MB/s | 23 MB 00:04 2023-09-30T00:59:40.456 INFO:teuthology.orchestra.run.smithi157.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.8 MB/s | 3.5 MB 00:00 2023-09-30T00:59:40.460 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:59:40.460 INFO:teuthology.orchestra.run.smithi157.stdout:Total 13 MB/s | 63 MB 00:04 2023-09-30T00:59:40.504 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T00:59:40.551 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T00:59:40.551 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T00:59:40.960 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T00:59:40.963 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T00:59:41.039 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T00:59:41.273 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:59:41.436 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:59:41.817 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:59:41.986 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:59:42.075 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:59:42.329 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:59:42.393 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T00:59:42.519 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:59:42.642 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:59:42.833 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:59:42.927 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:59:43.127 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:59:43.214 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:59:43.322 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:59:43.441 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:59:43.522 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:59:43.525 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:59:43.716 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:59:43.808 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:59:43.877 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:59:44.008 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:59:44.413 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:59:44.431 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:59:44.607 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:59:44.618 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:59:44.689 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:59:44.819 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:59:44.853 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:59:44.987 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:59:45.065 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:59:45.188 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:59:45.270 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:59:45.357 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:59:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:59:45.685 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:59:45.764 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:59:45.945 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:59:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:59:46.117 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:59:46.166 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:59:46.176 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:59:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:59:46.367 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:59:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:59:46.535 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:59:46.742 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:59:46.986 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:59:47.173 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:59:47.256 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:59:47.466 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:59:47.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:59:47.891 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:59:48.116 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:59:49.241 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:49.600 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:50.191 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:59:50.536 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:59:50.713 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:59:50.760 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:50.982 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:51.621 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:59:51.897 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:59:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:59:52.367 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:00:09.152 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:00:09.152 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2023-09-30T01:00:09.152 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2023-09-30T01:00:09.152 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-09-30T01:00:09.153 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:10.818 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:00:10.890 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:00:10.891 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:00:11.000 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:00:11.000 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2023-09-30T01:00:11.000 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2023-09-30T01:00:11.000 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-09-30T01:00:11.000 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /run 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-09-30T01:00:11.001 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:11.131 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:00:11.132 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:00:11.826 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T01:00:11.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T01:00:11.828 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:00:11.828 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:00:11.828 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:00:12.669 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:00:12.763 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:00:12.763 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.942 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T01:00:12.943 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:00:12.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:12.945 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:00:12.980 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:00:12.980 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:13.200 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2023-09-30T01:00:13.697 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:00:13.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:00:13.699 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:00:13.700 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:00:14.051 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:45 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.895 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T01:00:14.896 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:14.897 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:00:14.898 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:14.898 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:00:15.192 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2023-09-30T01:00:15.640 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:00:15.641 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:15.641 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:15.642 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 53 M 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 233 M 2023-09-30T01:00:15.643 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:00:16.044 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:45 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:00:17.048 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-09-30T01:00:17.173 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-09-30T01:00:17.249 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 436 kB/s | 175 kB 00:00 2023-09-30T01:00:17.307 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-09-30T01:00:17.441 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 368 kB/s | 71 kB 00:00 2023-09-30T01:00:17.641 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:00:17.642 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:17.642 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T01:00:17.643 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:17.644 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:00:17.644 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:17.644 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-09-30T01:00:17.644 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:17.644 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 53 M 2023-09-30T01:00:17.645 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 233 M 2023-09-30T01:00:17.645 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:00:18.199 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 992 kB/s | 202 kB 00:00 2023-09-30T01:00:18.257 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2023-09-30T01:00:18.341 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-09-30T01:00:18.400 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T01:00:18.433 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 398 kB/s | 175 kB 00:00 2023-09-30T01:00:24.224 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 8.3 MB/s | 52 MB 00:06 2023-09-30T01:00:24.226 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:00:24.227 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.0 MB/s | 53 MB 00:06 2023-09-30T01:00:24.270 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:00:24.292 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:00:24.292 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:00:24.607 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:00:24.610 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:00:25.108 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:00:25.332 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T01:00:25.562 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T01:00:25.726 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:00:25.795 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:00:25.983 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T01:00:26.133 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:00:28.889 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 4.3 MB/s | 52 MB 00:12 2023-09-30T01:00:28.891 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:00:28.891 INFO:teuthology.orchestra.run.smithi115.stdout:Total 4.0 MB/s | 53 MB 00:13 2023-09-30T01:00:28.934 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:00:28.955 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:00:28.955 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:00:29.264 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:00:29.267 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:00:29.760 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:00:30.025 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T01:00:30.219 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T01:00:30.367 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:00:30.476 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:00:30.657 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T01:00:30.799 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:00:31.107 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:00:31.365 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:00:31.641 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:00:31.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:00:31.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:00:31.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T01:00:31.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T01:00:31.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T01:00:32.412 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T01:00:32.412 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:32.412 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:00:32.412 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-7.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:32.413 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:00:32.624 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2023-09-30T01:00:33.484 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:00:35.041 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:00:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:00:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:00:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout:Install 32 Packages 2023-09-30T01:00:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:35.049 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 28 M 2023-09-30T01:00:35.049 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 110 M 2023-09-30T01:00:35.049 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:00:35.750 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2023-09-30T01:00:35.788 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:00:36.014 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:00:36.016 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 5.9 MB/s | 2.3 MB 00:00 2023-09-30T01:00:36.184 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.8 MB/s | 1.6 MB 00:00 2023-09-30T01:00:36.301 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 7.1 MB/s | 3.9 MB 00:00 2023-09-30T01:00:36.326 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 1.6 MB/s | 238 kB 00:00 2023-09-30T01:00:36.334 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:00:36.335 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:00:36.335 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:00:36.335 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T01:00:36.335 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T01:00:36.335 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T01:00:36.468 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 274 kB/s | 39 kB 00:00 2023-09-30T01:00:36.510 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 752 kB/s | 157 kB 00:00 2023-09-30T01:00:36.577 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 952 kB/s | 103 kB 00:00 2023-09-30T01:00:36.594 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 680 kB/s | 57 kB 00:00 2023-09-30T01:00:36.669 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2023-09-30T01:00:36.828 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2023-09-30T01:00:37.013 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-7.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:37.014 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:00:37.045 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 16 MB/s | 17 MB 00:01 2023-09-30T01:00:37.078 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 435 kB/s | 109 kB 00:00 2023-09-30T01:00:37.229 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-09-30T01:00:37.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2023-09-30T01:00:37.254 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2023-09-30T01:00:37.279 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 924 kB/s | 185 kB 00:00 2023-09-30T01:00:37.338 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2023-09-30T01:00:37.363 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 141 kB/s | 11 kB 00:00 2023-09-30T01:00:37.388 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 353 kB/s | 18 kB 00:00 2023-09-30T01:00:37.414 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 973 kB/s | 49 kB 00:00 2023-09-30T01:00:37.439 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-09-30T01:00:37.481 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2023-09-30T01:00:37.506 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-09-30T01:00:37.531 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-09-30T01:00:37.557 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2023-09-30T01:00:37.605 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 243 kB/s | 24 kB 00:00 2023-09-30T01:00:37.632 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-09-30T01:00:37.658 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-09-30T01:00:37.699 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-09-30T01:00:37.725 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2023-09-30T01:00:37.750 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-09-30T01:00:37.775 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-09-30T01:00:37.779 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:00:37.779 INFO:teuthology.orchestra.run.smithi157.stdout:Total 10 MB/s | 28 MB 00:02 2023-09-30T01:00:37.867 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:00:37.905 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:00:37.905 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:00:38.097 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:00:38.194 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:00:38.197 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:00:39.382 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:00:39.604 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T01:00:39.678 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:00:39.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:00:39.683 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:00:39.684 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:00:39.685 INFO:teuthology.orchestra.run.smithi115.stdout:Install 32 Packages 2023-09-30T01:00:39.686 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:39.686 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 28 M 2023-09-30T01:00:39.686 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 110 M 2023-09-30T01:00:39.686 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:00:39.856 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T01:00:40.074 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2023-09-30T01:00:40.107 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T01:00:40.330 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T01:00:40.527 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T01:00:40.757 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.8 MB/s | 2.3 MB 00:00 2023-09-30T01:00:40.766 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T01:00:40.993 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T01:00:41.246 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T01:00:41.513 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T01:00:41.624 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 956 kB/s | 1.6 MB 00:01 2023-09-30T01:00:41.792 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T01:00:41.808 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 3.9 MB 00:01 2023-09-30T01:00:41.892 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 893 kB/s | 238 kB 00:00 2023-09-30T01:00:42.049 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T01:00:42.066 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 222 kB/s | 39 kB 00:00 2023-09-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 538 kB/s | 157 kB 00:00 2023-09-30T01:00:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 622 kB/s | 57 kB 00:00 2023-09-30T01:00:42.217 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 687 kB/s | 103 kB 00:00 2023-09-30T01:00:42.299 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T01:00:42.435 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.8 MB/s | 457 kB 00:00 2023-09-30T01:00:42.468 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 993 kB/s | 249 kB 00:00 2023-09-30T01:00:42.535 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-09-30T01:00:42.669 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 327 kB/s | 44 kB 00:00 2023-09-30T01:00:42.689 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T01:00:42.761 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.9 MB/s | 641 kB 00:00 2023-09-30T01:00:42.786 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.5 MB/s | 185 kB 00:00 2023-09-30T01:00:42.936 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 985 kB/s | 173 kB 00:00 2023-09-30T01:00:42.962 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2023-09-30T01:00:42.987 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-09-30T01:00:42.992 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T01:00:43.012 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-09-30T01:00:43.038 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 970 kB/s | 49 kB 00:00 2023-09-30T01:00:43.080 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 876 kB/s | 59 kB 00:00 2023-09-30T01:00:43.122 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 375 kB/s | 16 kB 00:00 2023-09-30T01:00:43.164 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 451 kB/s | 19 kB 00:00 2023-09-30T01:00:43.203 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T01:00:43.212 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-09-30T01:00:43.239 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 318 kB/s | 24 kB 00:00 2023-09-30T01:00:43.264 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 567 kB/s | 29 kB 00:00 2023-09-30T01:00:43.281 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 657 kB/s | 27 kB 00:00 2023-09-30T01:00:43.472 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T01:00:43.490 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 6.1 MB/s | 17 MB 00:02 2023-09-30T01:00:43.515 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.0 MB/s | 255 kB 00:00 2023-09-30T01:00:43.557 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 624 kB/s | 23 kB 00:00 2023-09-30T01:00:43.732 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:00:43.791 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 494 kB/s | 251 kB 00:00 2023-09-30T01:00:43.816 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 262 kB/s | 85 kB 00:00 2023-09-30T01:00:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:00:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:Total 6.9 MB/s | 28 MB 00:04 2023-09-30T01:00:43.906 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:00:43.943 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:00:43.943 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:00:44.082 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T01:00:44.328 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T01:00:44.341 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:00:44.367 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:00:44.600 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T01:00:44.811 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T01:00:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T01:00:45.309 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T01:00:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T01:00:45.907 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:00:45.932 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T01:00:46.126 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T01:00:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T01:00:46.571 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T01:00:46.811 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T01:00:47.009 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T01:00:47.178 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T01:00:47.240 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T01:00:47.424 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T01:00:47.562 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T01:00:47.652 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T01:00:47.763 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:00:47.852 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T01:00:48.090 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T01:00:48.315 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T01:00:48.580 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T01:00:48.969 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T01:00:49.254 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:00:49.257 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T01:00:49.454 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T01:00:49.471 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:00:49.722 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T01:00:49.864 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:00:49.930 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:00:50.038 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:00:50.255 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:00:50.293 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T01:00:50.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:00:50.431 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:00:50.527 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T01:00:50.571 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:00:50.741 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T01:00:50.976 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T01:00:50.994 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T01:00:50.995 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T01:00:50.996 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T01:00:50.997 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T01:00:50.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T01:00:50.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T01:00:50.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T01:00:50.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T01:00:50.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T01:00:50.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T01:00:50.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T01:00:50.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T01:00:50.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T01:00:51.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T01:00:51.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T01:00:51.257 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T01:00:51.508 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T01:00:51.769 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T01:00:52.124 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T01:00:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T01:00:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:52.165 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:00:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T01:00:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:00:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:00:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:00:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T01:00:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:00:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:00:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:00:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:00:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:00:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:52.169 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:00:52.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2023-09-30T01:00:52.485 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T01:00:52.839 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T01:00:53.075 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:00:53.296 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:22 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:00:54.554 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:00:54.687 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:00:54.791 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:00:54.878 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:00:54.879 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:00:54.879 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:00:54.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2023-09-30T01:00:55.185 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:00:55.461 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:00:55.678 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:00:55.820 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:24 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:00:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:00:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:00:56.002 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:00:56.407 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T01:00:56.408 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T01:00:56.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T01:00:56.410 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T01:00:56.411 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T01:00:57.435 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:00:57.435 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:00:57.436 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 80 k 2023-09-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 363 k 2023-09-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:00:57.560 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T01:00:57.560 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T01:00:57.561 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:00:57.562 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:00:57.563 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:00:57.564 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:00:57.564 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:00:57.564 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:00:57.564 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:00:57.748 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2023-09-30T01:00:57.939 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 160 kB/s | 80 kB 00:00 2023-09-30T01:00:57.939 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:00:57.940 INFO:teuthology.orchestra.run.smithi157.stdout:Total 159 kB/s | 80 kB 00:00 2023-09-30T01:00:57.940 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:00:57.949 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:00:57.949 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:00:57.951 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:00:57.954 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:00:58.000 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:00:58.531 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:00:58.593 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:00:58.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:00:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:00:59.534 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:00:59.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:00:59.989 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:01:00.178 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:00.179 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:01:00.179 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:00.272 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2023-09-30T01:01:00.644 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:01.114 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:02.246 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:02.247 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 161 k 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 490 k 2023-09-30T01:01:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:02.551 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 534 kB/s | 161 kB 00:00 2023-09-30T01:01:02.552 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:02.552 INFO:teuthology.orchestra.run.smithi157.stdout:Total 531 kB/s | 161 kB 00:00 2023-09-30T01:01:02.552 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:02.563 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:02.563 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:02.610 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:02.613 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:02.702 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:01:02.703 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 80 k 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 363 k 2023-09-30T01:01:02.704 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:02.729 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:02.865 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 499 kB/s | 80 kB 00:00 2023-09-30T01:01:02.865 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:02.865 INFO:teuthology.orchestra.run.smithi115.stdout:Total 495 kB/s | 80 kB 00:00 2023-09-30T01:01:02.866 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:02.873 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:02.873 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:02.875 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:02.878 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:02.937 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:02.948 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:03.366 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:03.425 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:03.569 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:03.791 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:04.422 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:04.422 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:04.422 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:04.423 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:04.423 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:04.423 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:04.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:01:04.865 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:04.865 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:04.865 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:04.866 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:01:04.866 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:04.866 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:05.111 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2023-09-30T01:01:05.528 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:05.974 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:07.125 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:07.126 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:01:07.127 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:07.127 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 161 k 2023-09-30T01:01:07.127 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 490 k 2023-09-30T01:01:07.127 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:07.325 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:01:07.342 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 753 kB/s | 161 kB 00:00 2023-09-30T01:01:07.342 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:07.342 INFO:teuthology.orchestra.run.smithi115.stdout:Total 749 kB/s | 161 kB 00:00 2023-09-30T01:01:07.343 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:07.353 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:07.353 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:07.400 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:07.403 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:07.511 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:07.512 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:07.513 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:01:07.513 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:07.634 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:01:07.675 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:08.072 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:08.486 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:08.694 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:01:08.695 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:08.695 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:08.695 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:01:08.695 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:08.695 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:08.949 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2023-09-30T01:01:09.802 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:10.066 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:10.067 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:10.067 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:10.067 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:10.067 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-09-30T01:01:10.068 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:10.069 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2023-09-30T01:01:10.069 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 8.0 M 2023-09-30T01:01:10.069 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:10.670 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 49 kB/s | 14 kB 00:00 2023-09-30T01:01:10.771 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 58 kB/s | 23 kB 00:00 2023-09-30T01:01:10.812 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 301 kB/s | 43 kB 00:00 2023-09-30T01:01:10.880 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 310 kB/s | 34 kB 00:00 2023-09-30T01:01:11.013 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 983 kB/s | 196 kB 00:00 2023-09-30T01:01:11.198 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:01:11.382 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 1.7 MB/s | 1.7 MB 00:01 2023-09-30T01:01:11.384 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:11.384 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-09-30T01:01:11.385 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:11.386 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:01:11.386 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:11.396 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:11.406 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:11.406 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:11.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:01:11.494 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:11.497 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:11.783 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:11.990 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T01:01:12.223 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T01:01:12.325 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:12.395 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T01:01:12.549 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:01:12.864 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:01:13.100 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:01:13.393 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:01:13.393 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:01:13.394 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:01:13.394 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:01:13.394 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T01:01:13.394 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T01:01:13.908 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:13.909 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:01:13.910 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:13.911 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 2.0 M 2023-09-30T01:01:13.912 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 8.0 M 2023-09-30T01:01:13.912 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:01:14.029 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:01:14.030 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:01:14.030 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:14.030 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:14.145 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:01:14.564 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 170 kB/s | 23 kB 00:00 2023-09-30T01:01:14.631 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 71 kB/s | 14 kB 00:00 2023-09-30T01:01:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2023-09-30T01:01:14.749 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.3 MB/s | 1.7 MB 00:00 2023-09-30T01:01:14.974 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 98 kB/s | 34 kB 00:00 2023-09-30T01:01:14.993 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:15.182 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 413 kB/s | 196 kB 00:00 2023-09-30T01:01:15.184 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:15.184 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-09-30T01:01:15.196 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:15.207 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:15.207 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:15.288 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:15.291 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:15.581 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:15.771 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T01:01:16.033 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T01:01:16.218 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T01:01:16.346 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:01:16.571 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T01:01:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout:Install 8 Packages 2023-09-30T01:01:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:16.575 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 35 M 2023-09-30T01:01:16.575 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 210 M 2023-09-30T01:01:16.575 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:16.703 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:01:16.953 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:01:17.172 INFO:teuthology.orchestra.run.smithi157.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.5 MB/s | 4.9 MB 00:00 2023-09-30T01:01:17.245 INFO:teuthology.orchestra.run.smithi157.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2023-09-30T01:01:17.337 INFO:teuthology.orchestra.run.smithi157.stdout:(3/8): ceph-mgr-diskprediction-local-17.2.6-116 11 MB/s | 7.4 MB 00:00 2023-09-30T01:01:17.362 INFO:teuthology.orchestra.run.smithi157.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T01:01:17.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T01:01:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2023-09-30T01:01:17.663 INFO:teuthology.orchestra.run.smithi157.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 739 kB/s | 172 kB 00:00 2023-09-30T01:01:17.688 INFO:teuthology.orchestra.run.smithi157.stdout:(7/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-09-30T01:01:17.994 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:01:17.995 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:17.996 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:18.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:01:18.962 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:19.672 INFO:teuthology.orchestra.run.smithi157.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 5.9 MB/s | 14 MB 00:02 2023-09-30T01:01:19.672 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:19.672 INFO:teuthology.orchestra.run.smithi157.stdout:Total 11 MB/s | 35 MB 00:03 2023-09-30T01:01:19.911 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:19.929 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:19.930 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:20.547 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:20.548 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T01:01:20.548 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:20.548 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T01:01:20.548 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:20.548 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:20.549 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:20.550 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T01:01:20.550 INFO:teuthology.orchestra.run.smithi115.stdout:Install 8 Packages 2023-09-30T01:01:20.550 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:20.550 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 35 M 2023-09-30T01:01:20.550 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 210 M 2023-09-30T01:01:20.551 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:20.616 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:20.619 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:21.322 INFO:teuthology.orchestra.run.smithi115.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 13 MB/s | 7.4 MB 00:00 2023-09-30T01:01:21.522 INFO:teuthology.orchestra.run.smithi115.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.3 MB/s | 4.7 MB 00:00 2023-09-30T01:01:21.589 INFO:teuthology.orchestra.run.smithi115.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.0 MB/s | 4.9 MB 00:00 2023-09-30T01:01:21.645 INFO:teuthology.orchestra.run.smithi115.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2023-09-30T01:01:21.929 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:21.948 INFO:teuthology.orchestra.run.smithi115.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.0 MB/s | 3.7 MB 00:00 2023-09-30T01:01:21.973 INFO:teuthology.orchestra.run.smithi115.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-09-30T01:01:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:01:22.173 INFO:teuthology.orchestra.run.smithi115.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 766 kB/s | 172 kB 00:00 2023-09-30T01:01:22.253 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:01:22.399 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:01:22.415 INFO:teuthology.orchestra.run.smithi115.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 17 MB/s | 14 MB 00:00 2023-09-30T01:01:22.415 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:22.416 INFO:teuthology.orchestra.run.smithi115.stdout:Total 19 MB/s | 35 MB 00:01 2023-09-30T01:01:22.648 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:22.668 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:22.668 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:22.996 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:01:23.057 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:23.060 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:23.233 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:23.647 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:23.821 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:01:23.854 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:23.964 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:01:24.024 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:01:24.146 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:01:24.717 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:01:24.741 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:01:24.988 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:25.085 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:01:25.365 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:01:25.606 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:25.737 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:01:26.418 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:01:26.766 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:01:27.037 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:01:27.138 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:01:28.369 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T01:01:28.598 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:01:28.823 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:01:28.943 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T01:01:28.944 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T01:01:29.633 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T01:01:29.634 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:29.635 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:29.751 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:01:30.080 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T01:01:30.286 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:01:30.603 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:59 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T01:01:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:01:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T01:01:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T01:01:31.269 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T01:01:31.269 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:31.269 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:31.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:31.270 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:31.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:01:32.185 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:01:32.187 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:01:32.188 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:32.189 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:32.189 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:32.189 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2023-09-30T01:01:32.189 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:32.189 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.9 M 2023-09-30T01:01:32.190 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 25 M 2023-09-30T01:01:32.190 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:32.249 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:03 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:33.842 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:01:33.844 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T01:01:33.845 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2023-09-30T01:01:33.846 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:33.847 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.9 M 2023-09-30T01:01:33.847 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 25 M 2023-09-30T01:01:33.847 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:34.063 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 188 kB/s | 27 kB 00:00 2023-09-30T01:01:34.079 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 127 kB/s | 20 kB 00:00 2023-09-30T01:01:34.231 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 729 kB/s | 110 kB 00:00 2023-09-30T01:01:34.263 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 691 kB/s | 126 kB 00:00 2023-09-30T01:01:34.288 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 129 kB/s | 47 kB 00:00 2023-09-30T01:01:34.414 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 264 kB/s | 33 kB 00:00 2023-09-30T01:01:34.439 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 243 kB/s | 43 kB 00:00 2023-09-30T01:01:34.473 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 642 kB/s | 155 kB 00:00 2023-09-30T01:01:34.531 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2023-09-30T01:01:34.615 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 463 kB/s | 81 kB 00:00 2023-09-30T01:01:34.682 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 428 kB/s | 64 kB 00:00 2023-09-30T01:01:34.716 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 610 kB/s | 61 kB 00:00 2023-09-30T01:01:34.749 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.2 MB/s | 1.2 MB 00:00 2023-09-30T01:01:34.750 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:34.750 INFO:teuthology.orchestra.run.smithi157.stdout:Total 772 kB/s | 1.9 MB 00:02 2023-09-30T01:01:34.812 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:34.828 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:34.828 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:34.915 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:34.918 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:35.420 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:35.658 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:01:35.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T01:01:36.134 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T01:01:36.404 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T01:01:36.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T01:01:36.893 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T01:01:37.228 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T01:01:37.459 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:01:38.541 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:01:38.845 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T01:01:39.112 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:01:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 142 kB/s | 20 kB 00:00 2023-09-30T01:01:39.357 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 160 kB/s | 27 kB 00:00 2023-09-30T01:01:39.364 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T01:01:39.382 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 247 kB/s | 47 kB 00:00 2023-09-30T01:01:39.457 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-09-30T01:01:39.482 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 877 kB/s | 110 kB 00:00 2023-09-30T01:01:39.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:01:39.876 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:01:39.877 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T01:01:39.878 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T01:01:39.878 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T01:01:39.924 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 75 kB/s | 33 kB 00:00 2023-09-30T01:01:40.075 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 109 kB/s | 16 kB 00:00 2023-09-30T01:01:40.257 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 53 kB/s | 43 kB 00:00 2023-09-30T01:01:40.326 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 324 kB/s | 81 kB 00:00 2023-09-30T01:01:40.484 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 406 kB/s | 64 kB 00:00 2023-09-30T01:01:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T01:01:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:40.570 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:01:40.571 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:40.572 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:40.634 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 408 kB/s | 61 kB 00:00 2023-09-30T01:01:40.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:01:40.968 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 98 kB/s | 155 kB 00:01 2023-09-30T01:01:41.235 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.2 MB/s | 1.2 MB 00:00 2023-09-30T01:01:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:41.237 INFO:teuthology.orchestra.run.smithi115.stdout:Total 267 kB/s | 1.9 MB 00:07 2023-09-30T01:01:41.298 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:41.314 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:41.314 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:41.400 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:41.403 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:41.541 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:10 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:41.913 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:42.206 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:01:42.409 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T01:01:42.670 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T01:01:42.899 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T01:01:43.117 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T01:01:43.118 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T01:01:43.119 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout:Install 5 Packages 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:43.120 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.0 M 2023-09-30T01:01:43.121 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2023-09-30T01:01:43.121 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:43.363 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T01:01:43.506 INFO:teuthology.orchestra.run.smithi157.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 596 kB/s | 121 kB 00:00 2023-09-30T01:01:43.623 INFO:teuthology.orchestra.run.smithi157.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2023-09-30T01:01:43.671 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T01:01:43.850 INFO:teuthology.orchestra.run.smithi157.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 267 kB/s | 60 kB 00:00 2023-09-30T01:01:43.874 INFO:teuthology.orchestra.run.smithi157.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2023-09-30T01:01:43.921 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:01:44.107 INFO:teuthology.orchestra.run.smithi157.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 6.0 MB/s | 4.8 MB 00:00 2023-09-30T01:01:44.107 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:44.108 INFO:teuthology.orchestra.run.smithi157.stdout:Total 6.0 MB/s | 6.0 MB 00:00 2023-09-30T01:01:44.159 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:44.170 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:44.171 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:44.294 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:44.297 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:44.884 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T01:01:44.984 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:01:45.298 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T01:01:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:01:45.742 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T01:01:45.857 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:01:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:01:46.241 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T01:01:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T01:01:46.243 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T01:01:46.257 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T01:01:46.514 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T01:01:46.753 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:01:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T01:01:46.920 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:01:46.921 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:01:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:01:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:01:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:01:46.922 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:46.922 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:47.026 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:01:47.027 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:01:47.027 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T01:01:47.027 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T01:01:47.027 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:01:47.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:01:47.788 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:47.789 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:47.893 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:47.902 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2023-09-30T01:01:48.766 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:17 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:49.478 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:49.479 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:49.479 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:49.479 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:49.480 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:Install 5 Packages 2023-09-30T01:01:49.481 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 6.0 M 2023-09-30T01:01:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 26 M 2023-09-30T01:01:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:49.926 INFO:teuthology.orchestra.run.smithi115.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2023-09-30T01:01:49.951 INFO:teuthology.orchestra.run.smithi115.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 416 kB/s | 121 kB 00:00 2023-09-30T01:01:50.227 INFO:teuthology.orchestra.run.smithi115.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 8.5 MB/s | 4.8 MB 00:00 2023-09-30T01:01:50.352 INFO:teuthology.orchestra.run.smithi115.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 150 kB/s | 60 kB 00:00 2023-09-30T01:01:50.370 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:50.371 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:50.372 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2023-09-30T01:01:50.372 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:50.372 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 957 k 2023-09-30T01:01:50.372 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 3.0 M 2023-09-30T01:01:50.372 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:50.452 INFO:teuthology.orchestra.run.smithi115.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 905 kB/s | 476 kB 00:00 2023-09-30T01:01:50.453 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:50.453 INFO:teuthology.orchestra.run.smithi115.stdout:Total 6.1 MB/s | 6.0 MB 00:00 2023-09-30T01:01:50.506 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:50.517 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:50.517 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:50.624 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:50.627 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:50.639 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 365 kB/s | 83 kB 00:00 2023-09-30T01:01:50.715 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.8 MB/s | 874 kB 00:00 2023-09-30T01:01:50.715 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:50.715 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.7 MB/s | 957 kB 00:00 2023-09-30T01:01:50.726 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:50.747 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:50.747 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:50.835 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:50.838 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:50.969 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:51.005 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:51.273 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T01:01:51.321 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T01:01:51.478 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:01:51.912 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:01:51.912 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:01:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:01:52.534 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T01:01:52.535 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:52.535 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:52.535 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T01:01:52.535 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:52.535 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T01:01:52.900 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T01:01:53.073 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:01:53.387 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:01:53.388 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:01:53.388 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T01:01:53.388 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T01:01:53.388 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:01:53.822 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T01:01:54.041 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T01:01:54.042 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:01:54.042 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:54.042 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:54.187 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2023-09-30T01:01:54.696 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:23 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:55.044 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:01:56.290 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:01:56.290 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:56.290 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T01:01:56.290 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:01:56.291 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:56.292 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 267 k 2023-09-30T01:01:56.292 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2023-09-30T01:01:56.292 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:01:56.627 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 797 kB/s | 267 kB 00:00 2023-09-30T01:01:56.627 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:56.627 INFO:teuthology.orchestra.run.smithi157.stdout:Total 794 kB/s | 267 kB 00:00 2023-09-30T01:01:56.628 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:01:56.633 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:01:56.635 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 957 k 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 3.0 M 2023-09-30T01:01:56.636 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:01:56.638 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:01:56.638 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:01:56.675 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:01:56.678 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:01:56.863 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:01:56.876 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 415 kB/s | 83 kB 00:00 2023-09-30T01:01:57.076 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 874 kB 00:00 2023-09-30T01:01:57.077 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:01:57.077 INFO:teuthology.orchestra.run.smithi115.stdout:Total 2.1 MB/s | 957 kB 00:00 2023-09-30T01:01:57.087 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:01:57.108 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:01:57.108 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:01:57.119 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:57.192 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:01:57.195 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:01:57.365 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:01:57.538 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:57.678 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T01:01:57.880 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:01:58.148 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:01:58.148 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:58.148 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:01:58.149 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:01:58.149 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:01:58.149 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:01:58.301 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:01:58.302 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:01:58.378 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2023-09-30T01:01:58.930 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T01:01:58.930 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:58.930 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:01:58.931 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T01:01:58.931 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:01:58.931 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:01:59.116 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-volume 2023-09-30T01:01:59.253 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:28 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:01:59.981 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:30 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:00.832 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:00.833 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:02:00.834 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:00.834 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 130 k 2023-09-30T01:02:00.834 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 464 k 2023-09-30T01:02:00.834 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:02:01.034 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 652 kB/s | 130 kB 00:00 2023-09-30T01:02:01.034 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:01.034 INFO:teuthology.orchestra.run.smithi157.stdout:Total 647 kB/s | 130 kB 00:00 2023-09-30T01:02:01.035 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:02:01.040 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:02:01.040 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:02:01.089 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:02:01.092 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:02:01.199 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:02:01.492 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:01.555 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:01.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:01.557 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 267 k 2023-09-30T01:02:01.557 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.3 M 2023-09-30T01:02:01.557 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:01.811 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:01.993 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 612 kB/s | 267 kB 00:00 2023-09-30T01:02:01.994 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:01.994 INFO:teuthology.orchestra.run.smithi115.stdout:Total 610 kB/s | 267 kB 00:00 2023-09-30T01:02:01.994 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:02.004 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:02.005 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:02.043 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:02.046 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:02.226 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:02.440 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:02.480 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:02:02.645 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2023-09-30T01:02:02.892 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:03.477 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:03.518 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:32 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:03.643 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2023-09-30T01:02:04.507 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:04.914 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:05.107 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:05.108 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:05.108 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:05.207 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2023-09-30T01:02:06.068 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:06.073 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:06.074 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 130 k 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 464 k 2023-09-30T01:02:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 654 kB/s | 130 kB 00:00 2023-09-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:06.275 INFO:teuthology.orchestra.run.smithi115.stdout:Total 650 kB/s | 130 kB 00:00 2023-09-30T01:02:06.275 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:06.280 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:06.280 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:06.329 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:06.332 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:06.444 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:06.752 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:07.027 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:07.639 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:07.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:07.639 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:07.639 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:07.640 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:07.640 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:07.661 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:07.661 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:07.661 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T01:02:07.661 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:02:07.662 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 25 k 2023-09-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 109 k 2023-09-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:02:07.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2023-09-30T01:02:07.927 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 94 kB/s | 25 kB 00:00 2023-09-30T01:02:07.928 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:07.928 INFO:teuthology.orchestra.run.smithi157.stdout:Total 93 kB/s | 25 kB 00:00 2023-09-30T01:02:07.928 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:02:07.932 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:02:07.932 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:02:07.933 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:02:07.936 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:02:08.016 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:02:08.190 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:08.490 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:08.699 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:39 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:09.076 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:09.226 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2023-09-30T01:02:10.079 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:39 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:10.100 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:10.290 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:10.290 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:10.290 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:10.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2023-09-30T01:02:11.240 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:42 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:11.488 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:11.675 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:11.676 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:11.677 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:11.774 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2023-09-30T01:02:12.631 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:12.820 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:12.820 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:12.820 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:12.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:12.822 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 25 k 2023-09-30T01:02:12.822 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 109 k 2023-09-30T01:02:12.822 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:13.035 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 116 kB/s | 25 kB 00:00 2023-09-30T01:02:13.036 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:13.036 INFO:teuthology.orchestra.run.smithi115.stdout:Total 115 kB/s | 25 kB 00:00 2023-09-30T01:02:13.036 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:13.040 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:13.040 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:13.042 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:13.045 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:13.094 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:13.317 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:14.030 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:14.180 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:14.222 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:14.223 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:14.223 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:14.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2023-09-30T01:02:14.322 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2023-09-30T01:02:15.169 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:46 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:15.179 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:44 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:16.545 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:16.574 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:16.733 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:16.734 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:16.734 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:16.761 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:16.762 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:16.763 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:16.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2023-09-30T01:02:16.857 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2023-09-30T01:02:17.683 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:17.723 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:46 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:19.094 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:19.130 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:19.281 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:19.283 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:19.283 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:19.314 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:19.317 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:19.317 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:19.379 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2023-09-30T01:02:19.437 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2023-09-30T01:02:20.230 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:20.298 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:49 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:21.629 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:21.691 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:21.816 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:21.817 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:21.818 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:21.878 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:21.880 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:21.880 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:21.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2023-09-30T01:02:21.979 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2023-09-30T01:02:22.768 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:53 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:22.836 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:24.159 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:24.224 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:24.346 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:24.347 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:24.347 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:24.411 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:24.412 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T01:02:24.412 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:24.450 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2023-09-30T01:02:24.507 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2023-09-30T01:02:25.303 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:56 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:25.367 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:26.702 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:26.888 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:26.889 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:26.890 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:26.970 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:26.971 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:26.971 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T01:02:26.971 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:26.971 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:02:26.971 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 90 k 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 264 k 2023-09-30T01:02:26.972 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:02:26.988 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2023-09-30T01:02:27.192 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 410 kB/s | 90 kB 00:00 2023-09-30T01:02:27.192 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:27.192 INFO:teuthology.orchestra.run.smithi157.stdout:Total 408 kB/s | 90 kB 00:00 2023-09-30T01:02:27.193 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:02:27.202 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:02:27.202 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:02:27.250 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:02:27.254 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:02:27.356 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:02:27.629 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:27.840 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:58 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:27.919 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:28.557 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:28.557 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:28.557 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:02:28.557 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:28.557 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:28.558 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:28.759 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2023-09-30T01:02:29.234 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:02:29.424 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:29.426 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T01:02:29.426 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:29.526 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2023-09-30T01:02:29.629 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:58 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:30.376 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:03:01 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:31.203 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:02:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:31.205 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.5 M 2023-09-30T01:02:31.205 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 16 M 2023-09-30T01:02:31.205 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:02:31.711 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 6.8 MB/s | 3.5 MB 00:00 2023-09-30T01:02:31.711 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:31.712 INFO:teuthology.orchestra.run.smithi157.stdout:Total 6.8 MB/s | 3.5 MB 00:00 2023-09-30T01:02:31.712 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:02:31.723 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:02:31.723 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:02:31.786 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:02:31.789 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:02:31.964 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:31.965 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:31.965 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T01:02:31.965 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:31.965 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:31.965 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 90 k 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 264 k 2023-09-30T01:02:31.966 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:32.124 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 571 kB/s | 90 kB 00:00 2023-09-30T01:02:32.125 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:32.125 INFO:teuthology.orchestra.run.smithi115.stdout:Total 566 kB/s | 90 kB 00:00 2023-09-30T01:02:32.125 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:32.135 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:32.135 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:32.183 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:32.186 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:32.222 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:02:32.316 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:32.446 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:32.530 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:32.805 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:32.864 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:33.421 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:33.421 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:33.421 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:33.422 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:33.422 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:33.422 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:33.466 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:33.466 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:33.466 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:02:33.467 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:33.467 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:33.467 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:33.703 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2023-09-30T01:02:33.809 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2023-09-30T01:02:34.569 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:03 ago on Sat 30 Sep 2023 12:59:31 AM UTC. 2023-09-30T01:02:34.669 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:03:05 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:36.157 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T01:02:36.158 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T01:02:36.159 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:36.160 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 173 k 2023-09-30T01:02:36.160 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 552 k 2023-09-30T01:02:36.160 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T01:02:36.246 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:36.247 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:36.247 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T01:02:36.247 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.5 M 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 16 M 2023-09-30T01:02:36.248 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:36.346 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 929 kB/s | 173 kB 00:00 2023-09-30T01:02:36.347 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:36.347 INFO:teuthology.orchestra.run.smithi157.stdout:Total 923 kB/s | 173 kB 00:00 2023-09-30T01:02:36.347 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T01:02:36.358 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T01:02:36.358 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T01:02:36.407 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T01:02:36.410 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T01:02:36.548 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T01:02:36.728 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 7.2 MB/s | 3.5 MB 00:00 2023-09-30T01:02:36.729 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:36.729 INFO:teuthology.orchestra.run.smithi115.stdout:Total 7.2 MB/s | 3.5 MB 00:00 2023-09-30T01:02:36.730 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:36.736 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:36.741 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:36.741 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:36.803 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:36.806 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:37.009 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:37.247 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:37.430 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:37.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:37.661 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:37.661 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T01:02:37.661 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:37.662 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:37.662 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T01:02:37.889 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:37.908 DEBUG:teuthology.parallel:result is None 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:38.523 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:38.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2023-09-30T01:02:39.575 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:03:10 ago on Sat 30 Sep 2023 12:59:29 AM UTC. 2023-09-30T01:02:41.152 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T01:02:41.153 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:41.154 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 173 k 2023-09-30T01:02:41.154 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 552 k 2023-09-30T01:02:41.154 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T01:02:41.473 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 543 kB/s | 173 kB 00:00 2023-09-30T01:02:41.474 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:02:41.474 INFO:teuthology.orchestra.run.smithi115.stdout:Total 540 kB/s | 173 kB 00:00 2023-09-30T01:02:41.474 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T01:02:41.485 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T01:02:41.485 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T01:02:41.535 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T01:02:41.538 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T01:02:41.667 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T01:02:41.903 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:42.197 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:42.812 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:02:42.812 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:42.812 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T01:02:42.812 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:02:42.813 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:42.813 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T01:02:43.000 DEBUG:teuthology.parallel:result is None 2023-09-30T01:02:43.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:02:43.227 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T01:02:43.250 INFO:teuthology.orchestra.run.smithi115.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T01:02:43.250 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T01:02:43.251 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T01:02:43.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:02:43.393 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T01:02:43.421 INFO:teuthology.orchestra.run.smithi157.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T01:02:43.421 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T01:02:43.421 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T01:02:43.422 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T01:02:43.422 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:43.422 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:02:43.450 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:43.451 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:02:43.494 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T01:02:43.494 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:43.494 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:02:43.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:02:43.588 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:43.588 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:02:43.620 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:02:43.692 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T01:02:43.692 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:43.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:02:43.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:02:43.787 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:43.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:02:43.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:02:43.888 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T01:02:43.888 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:43.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:02:43.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:02:43.982 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:43.982 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:02:44.011 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:02:44.080 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T01:02:44.146 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T01:02:44.146 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2023-09-30T01:02:44.148 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2023-09-30T01:02:44.176 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T01:02:44.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:02:44.216 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:02:44.244 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T01:02:44.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:02:44.283 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:02:44.323 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T01:02:44.323 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T01:02:44.324 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406806', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk0GqMes8Qq3IX+gKj0dc7tzXhqKRmxhbppzXWrwEVrK/bWaCoqihwFWCgphmoKWY4NckYAsXjhGGM2ActeLA8=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFLTwF6gV082yTB3YMXebeTq3Z8SDVUY/zK81SDcvj45/0aCs9JXS8H7mD7/i7AtGoa7K0/sIYaPN1IHe50es0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T01:02:44.324 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:02:44.350 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:02:44.371 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:02:44.408 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:02:44.438 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:44.438 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:02:44.491 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:02:44.491 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 394315 Links: 1 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 01:02:37.636498726 +0000 2023-09-30T01:02:44.549 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 00:58:19.027913780 +0000 2023-09-30T01:02:44.550 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 00:58:19.027913780 +0000 2023-09-30T01:02:44.550 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T01:02:44.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:02:44.615 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T01:02:44.615 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T01:02:44.615 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000215868 s, 2.4 MB/s 2023-09-30T01:02:44.616 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:02:44.675 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 385717 Links: 1 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 01:02:37.636498726 +0000 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 00:58:19.572903572 +0000 2023-09-30T01:02:44.733 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 00:58:19.572903572 +0000 2023-09-30T01:02:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T01:02:44.734 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:02:44.799 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T01:02:44.799 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T01:02:44.799 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000206251 s, 2.5 MB/s 2023-09-30T01:02:44.800 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:02:44.859 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2023-09-30T01:02:44.917 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:02:44.917 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:44.917 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 392031 Links: 1 2023-09-30T01:02:44.917 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:44.917 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:44.918 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 01:02:37.636498726 +0000 2023-09-30T01:02:44.918 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 00:58:20.101893664 +0000 2023-09-30T01:02:44.918 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 00:58:20.101893664 +0000 2023-09-30T01:02:44.918 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T01:02:44.918 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:02:44.983 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T01:02:44.983 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T01:02:44.983 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000233494 s, 2.2 MB/s 2023-09-30T01:02:44.984 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:02:45.043 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 388914 Links: 1 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 01:02:37.636498726 +0000 2023-09-30T01:02:45.101 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 00:58:20.626883830 +0000 2023-09-30T01:02:45.102 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 00:58:20.626883830 +0000 2023-09-30T01:02:45.102 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T01:02:45.102 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:02:45.167 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T01:02:45.167 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T01:02:45.167 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00020953 s, 2.4 MB/s 2023-09-30T01:02:45.168 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:02:45.226 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:45.227 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:45.227 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:02:45.248 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:02:45.249 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2023-09-30T01:02:45.310 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:02:45.310 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 389080 Links: 1 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 01:02:32.616614630 +0000 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:58:12.640682866 +0000 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:58:12.640682866 +0000 2023-09-30T01:02:45.311 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T01:02:45.311 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:02:45.382 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T01:02:45.382 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T01:02:45.382 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000121847 s, 4.2 MB/s 2023-09-30T01:02:45.383 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:02:45.444 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 386739 Links: 1 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 01:02:32.616614630 +0000 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:58:13.196673141 +0000 2023-09-30T01:02:45.505 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:58:13.196673141 +0000 2023-09-30T01:02:45.506 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T01:02:45.506 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:02:45.572 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T01:02:45.573 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T01:02:45.573 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 8.4921e-05 s, 6.0 MB/s 2023-09-30T01:02:45.574 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:02:45.634 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2023-09-30T01:02:45.692 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 394360 Links: 1 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 01:02:32.616614630 +0000 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:58:13.762663240 +0000 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:58:13.762663240 +0000 2023-09-30T01:02:45.693 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T01:02:45.694 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:02:45.759 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T01:02:45.760 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T01:02:45.760 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000130952 s, 3.9 MB/s 2023-09-30T01:02:45.761 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:02:45.820 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 390938 Links: 1 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:02:45.878 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 01:02:32.616614630 +0000 2023-09-30T01:02:45.879 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:58:14.266654424 +0000 2023-09-30T01:02:45.879 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:58:14.266654424 +0000 2023-09-30T01:02:45.879 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T01:02:45.879 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:02:45.943 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T01:02:45.943 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T01:02:45.944 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000100014 s, 5.1 MB/s 2023-09-30T01:02:45.945 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:02:46.004 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:46.005 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T01:02:46.005 INFO:tasks.ceph:Generating config... 2023-09-30T01:02:46.006 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T01:02:46.006 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T01:02:46.006 INFO:tasks.ceph:[client] debug objecter = 20 2023-09-30T01:02:46.006 INFO:tasks.ceph:[client] debug rados = 20 2023-09-30T01:02:46.006 INFO:tasks.ceph:[global] lockdep = True 2023-09-30T01:02:46.006 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T01:02:46.006 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T01:02:46.007 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T01:02:46.007 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T01:02:46.007 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T01:02:46.007 INFO:tasks.ceph:[global] ms type = async 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T01:02:46.007 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T01:02:46.008 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T01:02:46.008 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T01:02:46.008 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-09-30T01:02:46.008 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T01:02:46.008 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd class default list = * 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd class load list = * 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T01:02:46.009 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T01:02:46.010 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T01:02:46.010 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T01:02:46.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T01:02:46.134 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T01:02:46.137 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T01:02:46.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:02:46.251 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.115'), ('mon.c', '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]'), ('mon.b', '172.21.15.157')] 2023-09-30T01:02:46.252 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T01:02:46.253 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:46.253 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:02:46.311 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.115 --addv c '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]' --add b 172.21.15.157 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: generated fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = octopus 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 0 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:02:46.397 INFO:teuthology.orchestra.run.smithi115.stdout:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:min_mon_release 15 (octopus) 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:election_strategy: 1 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:02:46.398 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T01:02:46.399 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:02:46.417 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c6b635fb-787f-4de6-b95e-9004242e56e2... 2023-09-30T01:02:46.418 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:02:46.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.502 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2023-09-30T01:02:46.503 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2023-09-30T01:02:46.504 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2023-09-30T01:02:46.505 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = "172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157" 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: lockdep = True 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: mon client directed command retry = 5 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: mon election default strategy = 1 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: ms bind msgr1 = False 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: ms bind msgr2 = True 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: ms inject socket failures = 5000 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: ms type = async 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:02:46.506 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.507 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:02:46.508 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2023-09-30T01:02:46.509 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression algorithm = zlib 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression mode = aggressive 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore zero block detection = True 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 20 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 20 2023-09-30T01:02:46.510 INFO:teuthology.orchestra.run.smithi115.stdout: debug monc = 20 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 10 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: osd class default list = * 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: osd class load list = * 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: osd client watch timeout = 120 2023-09-30T01:02:46.511 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify cached snaps = True 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify missing on start = True 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock override recovery settings = True 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue = debug_random 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue cut off = debug_random 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: osd shutdown pgref assert = True 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T01:02:46.512 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-09-30T01:02:46.513 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.514 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon min osdmap epochs = 50 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune interval = 2 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune min = 15 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:02:46.515 INFO:teuthology.orchestra.run.smithi115.stdout: mon scrub interval = 300 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on pool no app = False 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: paxos service trim min = 10 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:02:46.516 INFO:teuthology.orchestra.run.smithi115.stdout: debug monc = 20 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout: debug objecter = 20 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout: debug rados = 20 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.c] 2023-09-30T01:02:46.517 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.b] 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2023-09-30T01:02:46.518 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:02:46.519 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:02:46.520 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.521 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157" 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: lockdep = True 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: mon client directed command retry = 5 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 1 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: ms bind msgr1 = False 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: ms bind msgr2 = True 2023-09-30T01:02:46.522 INFO:teuthology.orchestra.run.smithi157.stdout: ms inject socket failures = 5000 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2023-09-30T01:02:46.523 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.524 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression algorithm = zlib 2023-09-30T01:02:46.525 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2023-09-30T01:02:46.526 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd class default list = * 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd class load list = * 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd client watch timeout = 120 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2023-09-30T01:02:46.527 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.528 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2023-09-30T01:02:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2023-09-30T01:02:46.530 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:02:46.530 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.530 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:02:46.530 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:02:46.530 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:02:46.531 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on pool no app = False 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2023-09-30T01:02:46.532 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T01:02:46.533 INFO:teuthology.orchestra.run.smithi157.stdout: debug objecter = 20 2023-09-30T01:02:46.534 INFO:teuthology.orchestra.run.smithi157.stdout: debug rados = 20 2023-09-30T01:02:46.534 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2023-09-30T01:02:46.534 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2023-09-30T01:02:46.534 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2023-09-30T01:02:46.535 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T01:02:46.536 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T01:02:46.628 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T01:02:46.629 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:46.629 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T01:02:46.647 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:46.647 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T01:02:46.704 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:02:46.705 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:46.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:02:46.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:02:46.781 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:02:46.781 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:46.839 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:02:46.839 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:46.839 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:02:46.839 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:02:46.878 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:02:46.878 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:46.936 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T01:02:46.936 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T01:02:46.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T01:02:46.986 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T01:02:46.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:02:47.064 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:02:47.066 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T01:02:47.066 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T01:02:47.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T01:02:47.107 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T01:02:47.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T01:02:47.162 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T01:02:47.174 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T01:02:47.174 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T01:02:47.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T01:02:47.202 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:47.202 INFO:tasks.ceph:role: osd.0 2023-09-30T01:02:47.202 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:02:47.203 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:02:47.277 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:02:47.277 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:02:47.277 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:02:47.277 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:02:47.278 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:02:47.716 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T01:02:47.717 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T01:02:47.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T01:02:47.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T01:02:47.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T01:02:47.993 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:47.993 INFO:tasks.ceph:role: osd.1 2023-09-30T01:02:47.993 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:02:47.994 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:02:48.068 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:02:48.068 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:02:48.068 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:02:48.069 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:02:48.493 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T01:02:48.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T01:02:48.494 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T01:02:48.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T01:02:48.635 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T01:02:48.703 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:48.703 INFO:tasks.ceph:role: osd.2 2023-09-30T01:02:48.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:02:48.703 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:02:48.777 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:02:48.778 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:02:49.224 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T01:02:49.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T01:02:49.226 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T01:02:49.386 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T01:02:49.420 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T01:02:49.487 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T01:02:49.487 INFO:tasks.ceph:role: osd.3 2023-09-30T01:02:49.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:02:49.488 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T01:02:49.563 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:02:49.563 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:02:49.563 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:02:49.563 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:02:49.563 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:02:49.564 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:02:49.564 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:02:49.564 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:02:49.564 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:02:49.564 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:02:50.008 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T01:02:50.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T01:02:50.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T01:02:50.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T01:02:50.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:50.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.366+0000 7f3d4af89540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T01:02:50.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.367+0000 7f3d4af89540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T01:02:50.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.367+0000 7f3d4af89540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:02:50.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.367+0000 7f3d4af89540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:02:50.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.367+0000 7f3d4af89540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:02:50.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.367+0000 7f3d4af89540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T01:02:50.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:50.891+0000 7f3d4af89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:51.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:51.654+0000 7f3d4af89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:52.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:52.418+0000 7f3d4af89540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:53.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T01:02:53.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:53.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.337+0000 7fa4ff0c7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T01:02:53.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.337+0000 7fa4ff0c7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T01:02:53.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.337+0000 7fa4ff0c7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:02:53.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.337+0000 7fa4ff0c7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:02:53.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.337+0000 7fa4ff0c7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:02:53.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.338+0000 7fa4ff0c7540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T01:02:53.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:53.862+0000 7fa4ff0c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:54.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:54.631+0000 7fa4ff0c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:55.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:55.393+0000 7fa4ff0c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:56.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T01:02:56.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:56.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.311+0000 7fc34758d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T01:02:56.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.311+0000 7fc34758d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T01:02:56.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.311+0000 7fc34758d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:02:56.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.311+0000 7fc34758d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:02:56.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.311+0000 7fc34758d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:02:56.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.312+0000 7fc34758d540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T01:02:56.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:56.836+0000 7fc34758d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:57.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:57.601+0000 7fc34758d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:58.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:58.370+0000 7fc34758d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:02:59.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T01:02:59.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:02:59.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T01:02:59.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T01:02:59.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:02:59.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:02:59.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:02:59.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.278+0000 7f8aa0224540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T01:02:59.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:02:59.800+0000 7f8aa0224540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:00.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:00.568+0000 7f8aa0224540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:01.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:01.330+0000 7f8aa0224540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:02.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T01:03:02.156 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T01:03:02.190 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:03:02.191 INFO:tasks.ceph:role: osd.4 2023-09-30T01:03:02.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:03:02.191 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:03:02.261 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:03:02.262 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:03:02.285 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T01:03:02.286 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T01:03:02.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T01:03:02.369 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T01:03:02.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T01:03:02.511 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:03:02.511 INFO:tasks.ceph:role: osd.5 2023-09-30T01:03:02.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:03:02.512 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:03:02.579 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:03:02.579 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:03:02.579 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:03:02.579 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:03:02.579 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:03:02.580 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:03:02.580 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:03:02.580 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:03:02.580 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:03:02.580 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:03:02.601 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T01:03:02.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T01:03:02.603 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T01:03:02.683 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T01:03:02.755 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T01:03:02.823 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:03:02.823 INFO:tasks.ceph:role: osd.6 2023-09-30T01:03:02.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:03:02.823 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:03:02.889 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:03:02.889 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:03:02.890 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:03:02.912 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T01:03:02.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T01:03:02.913 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T01:03:02.993 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T01:03:03.064 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T01:03:03.132 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T01:03:03.132 INFO:tasks.ceph:role: osd.7 2023-09-30T01:03:03.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:03:03.133 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T01:03:03.199 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:03:03.200 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:03:03.222 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T01:03:03.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T01:03:03.223 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T01:03:03.304 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T01:03:03.375 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:03.468 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T01:03:03.468 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T01:03:03.468 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:03:03.469 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:03:03.469 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:03:03.469 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.468+0000 7f5265f3c540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T01:03:03.991 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:03.991+0000 7f5265f3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:04.757 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:04.757+0000 7f5265f3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:05.520 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:05.520+0000 7f5265f3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:06.317 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T01:03:06.350 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:06.438 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T01:03:06.438 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T01:03:06.438 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:03:06.439 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:03:06.439 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:03:06.439 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.438+0000 7f60e4fc4540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T01:03:06.967 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:06.967+0000 7f60e4fc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:07.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:07.726+0000 7f60e4fc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:08.490 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:08.490+0000 7f60e4fc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:09.284 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T01:03:09.317 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:09.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.404+0000 7fa396b07540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T01:03:09.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.404+0000 7fa396b07540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T01:03:09.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.404+0000 7fa396b07540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:03:09.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.404+0000 7fa396b07540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:03:09.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.404+0000 7fa396b07540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T01:03:09.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.405+0000 7fa396b07540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T01:03:09.928 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:09.927+0000 7fa396b07540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:10.690 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:10.689+0000 7fa396b07540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:11.457 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:11.456+0000 7fa396b07540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:12.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T01:03:12.282 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.372+0000 7ff101bb7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.373+0000 7ff101bb7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.373+0000 7ff101bb7540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.373+0000 7ff101bb7540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.373+0000 7ff101bb7540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T01:03:12.374 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.373+0000 7ff101bb7540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T01:03:12.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:12.895+0000 7ff101bb7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:13.662 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:13.661+0000 7ff101bb7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:14.427 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:14.426+0000 7ff101bb7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:03:15.220 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T01:03:15.253 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T01:03:15.253 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.254 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T01:03:15.286 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.286 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T01:03:15.358 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.358 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T01:03:15.427 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.427 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T01:03:15.497 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T01:03:15.565 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.565 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T01:03:15.598 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T01:03:15.670 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T01:03:15.740 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.741 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T01:03:15.809 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.810 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T01:03:15.878 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:15.879 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T01:03:15.898 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:15.898 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T01:03:15.937 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T01:03:15.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:03:15.940 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:03:15.981 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.y] 2023-09-30T01:03:15.981 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC2cxdlEjThOhAAg1vx2PA+PIe5B9UPRsIK5Q== 2023-09-30T01:03:15.981 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.0] 2023-09-30T01:03:15.981 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC6cxdlarHtFRAAaPPm2ru1VTWetXoOsrvgcw== 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.1] 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC9cxdlhl8tFBAAIghQXx8LTFLOcnJtG3u+rg== 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.2] 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDAcxdlDcucEhAAXVDcgxm4kL1xnU+A0ka2Ew== 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.3] 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDDcxdl6o6iEBAAzBD+w2IysQuekuxC5sCNNQ== 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.x] 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC3cxdloPPdAxAA6sthWIZ44RuH9eQHKSi55g== 2023-09-30T01:03:15.982 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.4] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDHcxdluxPzGxAAHhn6dK9i/KsRAZT4rWVnnQ== 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.5] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDKcxdlBrgrGhAAdERX6WQ0Tur3Z5qodf4/Dw== 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.6] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDNcxdlMEU4GBAAelEsEsPGmKcoIDayJ5hDEQ== 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.7] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDQcxdloCpUFhAAQ2boF2j/50hur8E4toFBZA== 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC3cxdlPhlzBhAAOkDFWV51x8kBz8HeMPaegg== 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2023-09-30T01:03:15.983 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC3cxdlZkG7CRAACrmxpW8AVJEPg8JH7aDweg== 2023-09-30T01:03:16.004 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2023-09-30T01:03:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC2cxdlEjThOhAAg1vx2PA+PIe5B9UPRsIK5Q== 2023-09-30T01:03:16.004 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-09-30T01:03:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC6cxdlarHtFRAAaPPm2ru1VTWetXoOsrvgcw== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC9cxdlhl8tFBAAIghQXx8LTFLOcnJtG3u+rg== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDAcxdlDcucEhAAXVDcgxm4kL1xnU+A0ka2Ew== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDDcxdl6o6iEBAAzBD+w2IysQuekuxC5sCNNQ== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3cxdloPPdAxAA6sthWIZ44RuH9eQHKSi55g== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDHcxdluxPzGxAAHhn6dK9i/KsRAZT4rWVnnQ== 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2023-09-30T01:03:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDKcxdlBrgrGhAAdERX6WQ0Tur3Z5qodf4/Dw== 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDNcxdlMEU4GBAAelEsEsPGmKcoIDayJ5hDEQ== 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDQcxdloCpUFhAAQ2boF2j/50hur8E4toFBZA== 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3cxdlPhlzBhAAOkDFWV51x8kBz8HeMPaegg== 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2023-09-30T01:03:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3cxdlZkG7CRAACrmxpW8AVJEPg8JH7aDweg== 2023-09-30T01:03:16.007 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:03:16.027 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:03:16.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.102 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.154 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.206 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.255 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.308 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:03:16.310 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:03:16.360 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.362 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.411 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.464 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.466 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.518 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:03:16.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:03:16.572 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:03:16.626 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:03:16.628 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:03:16.682 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T01:03:16.682 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T01:03:16.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:03:16.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T01:03:16.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T01:03:16.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:03:16.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T01:03:17.018 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T01:03:17.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:03:17.142 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T01:03:17.169 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:17.171 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:03:17.226 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T01:03:17.227 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T01:03:17.227 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T01:03:17.229 INFO:tasks.ceph.mon.a:Started 2023-09-30T01:03:17.229 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T01:03:17.229 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T01:03:17.231 INFO:tasks.ceph.mon.c:Started 2023-09-30T01:03:17.231 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T01:03:17.231 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T01:03:17.269 INFO:tasks.ceph.mon.b:Started 2023-09-30T01:03:17.269 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T01:03:17.269 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T01:03:17.269 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T01:03:17.272 INFO:tasks.ceph.mgr.y:Started 2023-09-30T01:03:17.272 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T01:03:17.272 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T01:03:17.274 INFO:tasks.ceph.mgr.x:Started 2023-09-30T01:03:17.274 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T01:03:17.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T01:03:17.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.401+0000 7f32bc8ec700 10 monclient: get_monmap_and_config 2023-09-30T01:03:17.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.401+0000 7f32bc8ec700 10 monclient: build_initial_monmap 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.401+0000 7f32bc8ec700 10 monclient: monmap: 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:17.402516+0000 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:17.402516+0000 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:17.403 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:17.403 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 Processor -- start 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 -- start start 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient: init 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T01:03:17.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-b con 0x7f32b80ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-c con 0x7f32b80aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-a con 0x7f32b81415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b80a7040 con 0x7f32b81415c0 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b80b19b0 con 0x7f32b80aecc0 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:17.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a57f0 con 0x7f32b80ae8f0 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.402+0000 7f32bc8ec700 10 monclient(hunting): authenticate will time out at 1058.852975s 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58146/0 (socket says 172.21.15.115:58146) 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:35382/0 (socket says 172.21.15.115:35382) 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 learned_addr learned my addr 172.21.15.115:0/771152126 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f32b80ae8f0 auth_method 0 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:17.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f32b81415c0 auth_method 0 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f32b80aecc0 auth_method 0 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.403+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:03:17.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f32b81415c0 auth_method 0 2023-09-30T01:03:17.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f32b80ae8f0 auth_method 0 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f32b80aecc0 auth_method 0 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:17.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:17.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:17.604+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:03:18.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f32b81415c0 auth_method 0 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f32b80ae8f0 auth_method 0 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:03:18.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f32b80aecc0 auth_method 0 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.005+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:03:18.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f32b80ae8f0 auth_method 0 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f32b81415c0 auth_method 0 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f32b80aecc0 auth_method 0 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:18.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:18.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:18.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:03:18.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:18.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:18.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:18.806+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:03:20.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient(hunting): tick 2023-09-30T01:03:20.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 monclient(hunting): continuing hunt 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 msgr2=0x7f32b80a5d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32b80ae8f0 0x7f32b80a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 msgr2=0x7f32b80b1470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.157:3300/0 conn(0x7f32b80aecc0 0x7f32b80b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 -- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 --2- 172.21.15.115:0/771152126 >> v2:172.21.15.115:3300/0 conn(0x7f32b81415c0 0x7f32b80adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient: _add_conns ranks=[1,0] 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32a0003700 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:20.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32a0007380 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:20.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 -- 172.21.15.115:0/771152126 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b80a5480 con 0x7f32a0007380 2023-09-30T01:03:20.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 1 -- 172.21.15.115:0/771152126 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a000c0a0 con 0x7f32a0003700 2023-09-30T01:03:20.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:20.403+0000 7f32a7fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:22.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145992919 0 0) 0x7f32b80a5480 con 0x7f32a0007380 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 10 monclient(hunting): my global_id is 4098 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3298003570 con 0x7f32a0007380 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.576+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517448963 0 0) 0x7f32a000c0a0 con 0x7f32a0003700 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): my global_id is 4102 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b80a5480 con 0x7f32a0003700 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 178438726 0 0) 0x7f32b80a5480 con 0x7f32a0003700 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 >> v1:172.21.15.115:6789/0 conn(0x7f32a0007380 legacy=0x7f32a00097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: _finish_hunting 0 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 monclient: found mon.noname-c-legacy 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.157:6789/0 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a000d2a0 con 0x7f32a0003700 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: _finish_auth 0 2023-09-30T01:03:22.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f32a800c2f0 con 0x7f32a0003700 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32bc8ec700 5 monclient: authenticate success, global_id 4102 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32bc8ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: dump: 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.585 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] but i'm connected to v1:172.21.15.157:6789/0, reconnecting 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 >> v1:172.21.15.157:6789/0 conn(0x7f32a0003700 legacy=0x7f32a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 reap_dead start 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): picked mon.a con 0x7f3298003c90 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): picked mon.c con 0x7f3298004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): picked mon.b con 0x7f3298009a50 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329800c3c0 con 0x7f3298003c90 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329800c530 con 0x7f3298009a50 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329800c6f0 con 0x7f3298004810 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b559a700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32bc8ec700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f3298003c90 auth_method 0 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f3298004810 auth_method 0 2023-09-30T01:03:22.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f3298009a50 auth_method 0 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:22.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.577+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 msgr2=0x7f3298004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 msgr2=0x7f329800be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:22.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 monclient: found mon.a 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80a5480 con 0x7f3298003c90 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient: _finish_auth 0 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009060 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14169ef562142ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 msgr2=0x7f3298004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 msgr2=0x7f3298004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009060 tx=0x7f32a8008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009060 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient: ms_handle_reset current mon [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:22.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 msgr2=0x7f3298004060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298003c90 0x7f3298004060 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 0x7f329800ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): picked mon.a con 0x7f32a0003700 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): picked mon.b con 0x7f3298009a50 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): picked mon.c con 0x7f3298004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329800f6b0 con 0x7f32a0003700 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329800f7f0 con 0x7f3298009a50 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329800f930 con 0x7f3298004810 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b559a700 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f3298004810 auth_method 0 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 0x7f329800ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f32a0003700 auth_method 0 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 msgr2=0x7f329800be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b5d9b700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 msgr2=0x7f329800be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.578+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 msgr2=0x7f329800be30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 msgr2=0x7f329800ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 0x7f329800ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 monclient: found mon.c 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:22.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 -- 172.21.15.115:0/771152126 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f32a000c0a0 con 0x7f3298004810 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 10 monclient: _finish_auth 0 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b659c700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 0x7f329800ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.579+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f32b80a62b0 tx=0x7f32b000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89dd4e7619216911 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0016b30 con 0x7f3298004810 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 10 monclient: dump: 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.596 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 1 -- 172.21.15.115:0/771152126 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0016c90 con 0x7f3298004810 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32bc8ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32b559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32bc8ec700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.581+0000 7f32bc8ec700 4 set_mon_vals no callback set 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 10 monclient: shutdown 2023-09-30T01:03:22.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 -- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 msgr2=0x7f3298004be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f32b80a62b0 tx=0x7f32b000b040 comp rx=0 tx=0).stop 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 -- 172.21.15.115:0/771152126 shutdown_connections 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3298004810 0x7f3298004be0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f329800be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/771152126 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32a0003700 0x7f329800ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 -- 172.21.15.115:0/771152126 >> 172.21.15.115:0/771152126 conn(0x7f32b806a460 msgr2=0x7f32b80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 -- 172.21.15.115:0/771152126 shutdown_connections 2023-09-30T01:03:22.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.582+0000 7f32bc8ec700 1 -- 172.21.15.115:0/771152126 wait complete. 2023-09-30T01:03:22.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.583+0000 7f32bc8ec700 10 monclient: build_initial_monmap 2023-09-30T01:03:22.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.583+0000 7f32bc8ec700 10 monclient: monmap: 2023-09-30T01:03:22.602 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:22.602 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:22.602 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.583+0000 7f32bc8ec700 1 librados: starting msgr at 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.583+0000 7f32bc8ec700 1 librados: starting objecter 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 Processor -- start 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 -- start start 2023-09-30T01:03:22.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 librados: setting wanted keys 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 librados: calling monclient init 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient: init 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-a con 0x7f3298004810 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f32b813e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-b con 0x7f3298009a50 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32a0003700 0x7f32b80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): picked mon.noname-c con 0x7f32a0003700 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:22.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a8b60 con 0x7f3298004810 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b813e260 con 0x7f3298009a50 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b813d1d0 con 0x7f32a0003700 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32bc8ec700 10 monclient(hunting): authenticate will time out at 1064.034734s 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57612/0 (socket says 172.21.15.115:57612) 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 1 -- 172.21.15.115:0/3923811171 learned_addr learned my addr 172.21.15.115:0/3923811171 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): get_auth_request con 0x7f3298004810 auth_method 0 2023-09-30T01:03:22.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f32b813e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request con 0x7f3298009a50 auth_method 0 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b6d9d700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32a0003700 0x7f32b80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request con 0x7f32a0003700 auth_method 0 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:22.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.584+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 10 monclient(hunting): handle_auth_done global_id 4110 payload 824 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 -- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32a0003700 msgr2=0x7f32b80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32a0003700 0x7f32b80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 -- 172.21.15.115:0/3923811171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 msgr2=0x7f32b813e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f32b813e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 monclient: found mon.noname-a 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 -- 172.21.15.115:0/3923811171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b813ddf0 con 0x7f3298004810 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b5d9b700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f32b813e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b659c700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f329800c500 tx=0x7f32a8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 client.?.objecter ms_handle_connect 0x7f3298004810 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a800c2f0 con 0x7f3298004810 2023-09-30T01:03:22.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: dump: 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:22.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a8026050 con 0x7f3298004810 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 5 monclient: authenticate success, global_id 4110 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8020450 con 0x7f3298004810 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 10 monclient: _renew_subs 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b813df80 con 0x7f3298004810 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 monclient: dump: 2023-09-30T01:03:22.610 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32b4d99700 4 set_mon_vals no callback set 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:22.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 10 monclient: _renew_subs 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b80a7970 con 0x7f3298004810 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32bc8ec700 1 librados: init done 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f32a8020ac0 con 0x7f3298004810 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f32a80359a0 con 0x7f3298004810 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 10 client.4110.objecter ms_dispatch 0x7f32b8001550 osd_map(1..1 src has 1..1) v4 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 3 client.4110.objecter handle_osd_map decoding full epoch 1 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.585+0000 7f32a67fc700 20 client.4110.objecter dump_active .. 0 homeless 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.586+0000 7f32bc8ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.586+0000 7f32bc8ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.586+0000 7f32bc8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.586+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8072960 con 0x7f3298004810 2023-09-30T01:03:22.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.591+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f32a803a060 con 0x7f3298004810 2023-09-30T01:03:22.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.591+0000 7f32a67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:22.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.591+0000 7f32a67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:22.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.668+0000 7f32bc8ec700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:03:22.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.668+0000 7f32bc8ec700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:03:22.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.668+0000 7f32bc8ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:22.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.668+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f32b813fa40 con 0x7f3298004810 2023-09-30T01:03:22.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.674+0000 7f32a67fc700 1 -- 172.21.15.115:0/3923811171 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f32a8030070 con 0x7f3298004810 2023-09-30T01:03:22.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.674+0000 7f32a67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:03:22.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.674+0000 7f32a67fc700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T01:03:22.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 10 librados: watch_flush enter 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 10 librados: watch_flush exit 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 20 client.4110.objecter shutdown clearing up homeless session... 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 10 client.4110.objecter successfully canceled tick 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 10 monclient: shutdown 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 msgr2=0x7f32b80b3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f329800c500 tx=0x7f32a8020000 comp rx=0 tx=0).stop 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 shutdown_connections 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f32a0003700 0x7f32b80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3298004810 0x7f32b80b3f00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 --2- 172.21.15.115:0/3923811171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298009a50 0x7f32b813e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 >> 172.21.15.115:0/3923811171 conn(0x7f32b806a460 msgr2=0x7f32b80adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 shutdown_connections 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 -- 172.21.15.115:0/3923811171 wait complete. 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:22.675+0000 7f32bc8ec700 1 librados: shutdown 2023-09-30T01:03:22.677 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2023-09-30T01:03:22.687 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T01:03:22.687 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:22.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T01:03:22.711 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:22.710+0000 7fa6f95401c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:03:22.714 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:22.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T01:03:22.720 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:22.720+0000 7fa3f6d3d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:03:22.747 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:22.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T01:03:22.817 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T01:03:22.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T01:03:22.876 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:22.875+0000 7fa6f95401c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T01:03:22.892 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:22.893 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T01:03:22.900 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:22.900+0000 7fa3f6d3d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T01:03:22.919 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:22.920 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T01:03:22.987 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:22.987 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T01:03:23.054 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T01:03:23.054 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T01:03:23.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new f8068422-b1d0-4236-a4eb-fa8aecefe228 0 2023-09-30T01:03:23.249 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:23.249+0000 7fa3f6d3d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:03:23.253 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:23.253+0000 7fa6f95401c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:03:23.280 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.279+0000 7f3661e3b700 10 monclient: get_monmap_and_config 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.279+0000 7f3661e3b700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.279+0000 7f3661e3b700 10 monclient: monmap: 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:23.281621+0000 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:23.281621+0000 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:23.281 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:23.282 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:23.282 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.291 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.280+0000 7f3661e3b700 1 Processor -- start 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.280+0000 7f3661e3b700 1 -- start start 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.280+0000 7f3661e3b700 10 monclient: init 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f365c0a5d10 0x7f365c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-c con 0x7f365c0a5d10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0a6190 con 0x7f365c0ae8f0 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0a7040 con 0x7f365c0aecc0 2023-09-30T01:03:23.292 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365c141cd0 con 0x7f365c0a5d10 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f3661e3b700 10 monclient(hunting): authenticate will time out at 1048.745503s 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f365c0a5d10 0x7f365c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365affd700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f365c0aecc0 0x7f365c1415c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44546/0 (socket says 172.21.15.157:44546) 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365affd700 1 -- 172.21.15.157:0/2517185335 learned_addr learned my addr 172.21.15.157:0/2517185335 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): get_auth_request con 0x7f365c0a5d10 auth_method 0 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.293 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 1 -- 172.21.15.157:0/2517185335 >> v1:172.21.15.157:6789/0 conn(0x7f365c0aecc0 legacy=0x7f365c1415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 1 -- 172.21.15.157:0/2517185335 >> v1:172.21.15.115:6789/0 conn(0x7f365c0ae8f0 legacy=0x7f365c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 1 monclient: found mon.noname-c 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.281+0000 7f365bfff700 1 -- 172.21.15.157:0/2517185335 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c0a57f0 con 0x7f365c0a5d10 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365bfff700 1 --2- 172.21.15.157:0/2517185335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0a5d10 0x7f365c0adb60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f36540098e0 tx=0x7f36540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4ebc4f45202ea7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.294 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 1 -- 172.21.15.157:0/2517185335 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654009e80 con 0x7f365c0a5d10 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: dump: 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.295 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 5 monclient: authenticate success, global_id 4114 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 1 -- 172.21.15.157:0/2517185335 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365400b860 con 0x7f365c0a5d10 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 4 set_mon_vals no callback set 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 1 -- 172.21.15.157:0/2517185335 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400bbe0 con 0x7f365c0a5d10 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f365a7fc700 10 monclient: dump: 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.297 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 10 monclient: shutdown 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 1 -- 172.21.15.157:0/2517185335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0a5d10 msgr2=0x7f365c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 1 --2- 172.21.15.157:0/2517185335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0a5d10 0x7f365c0adb60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f36540098e0 tx=0x7f36540091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 1 -- 172.21.15.157:0/2517185335 shutdown_connections 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 1 --2- 172.21.15.157:0/2517185335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0a5d10 0x7f365c0adb60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.282+0000 7f3661e3b700 1 -- 172.21.15.157:0/2517185335 >> 172.21.15.157:0/2517185335 conn(0x7f365c06a460 msgr2=0x7f365c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.283+0000 7f3661e3b700 1 -- 172.21.15.157:0/2517185335 shutdown_connections 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.283+0000 7f3661e3b700 1 -- 172.21.15.157:0/2517185335 wait complete. 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.283+0000 7f3661e3b700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.283+0000 7f3661e3b700 10 monclient: monmap: 2023-09-30T01:03:23.298 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 librados: starting msgr at 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 librados: starting objecter 2023-09-30T01:03:23.299 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 Processor -- start 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 -- start start 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 librados: setting wanted keys 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 1 librados: calling monclient init 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.284+0000 7f3661e3b700 10 monclient: init 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 0x7f365c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-a con 0x7f365c0a5d10 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:23.300 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-b con 0x7f365c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 0x7f365c151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): picked mon.noname-c con 0x7f365c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.302 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c0aa480 con 0x7f365c0a5d10 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c151c80 con 0x7f365c0ae8f0 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c0aba00 con 0x7f365c0aecc0 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f3661e3b700 10 monclient(hunting): authenticate will time out at 1048.749319s 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57296/0 (socket says 172.21.15.157:57296) 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 1 -- 172.21.15.157:0/3887958335 learned_addr learned my addr 172.21.15.157:0/3887958335 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): get_auth_request con 0x7f365c0ae8f0 auth_method 0 2023-09-30T01:03:23.303 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 0x7f365c151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 0x7f365c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): get_auth_request con 0x7f365c0aecc0 auth_method 0 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): get_auth_request con 0x7f365c0a5d10 auth_method 0 2023-09-30T01:03:23.304 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.305 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 -- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 msgr2=0x7f365c151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 0x7f365c151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 -- 172.21.15.157:0/3887958335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 msgr2=0x7f365c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365b7fe700 1 -- 172.21.15.157:0/3887958335 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654009420 con 0x7f365c0a5d10 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365bfff700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 0x7f365c151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:23.306 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.285+0000 7f365affd700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f365b7fe700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 0x7f365c13e480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f364c00c940 tx=0x7f364c0016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 client.?.objecter ms_handle_connect 0x7f365c0a5d10 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c012630 con 0x7f365c0a5d10 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: dump: 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.307 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364c0127b0 con 0x7f365c0a5d10 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 5 monclient: authenticate success, global_id 4116 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 10 monclient: _renew_subs 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:23.308 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c0aa910 con 0x7f365c0a5d10 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c012b70 con 0x7f365c0a5d10 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3659ffb700 4 set_mon_vals no callback set 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 monclient: dump: 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.309 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 10 monclient: _renew_subs 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c0a9ab0 con 0x7f365c0a5d10 2023-09-30T01:03:23.310 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f3661e3b700 1 librados: init done 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f364c003030 con 0x7f365c0a5d10 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f364c01be20 con 0x7f365c0a5d10 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 10 client.4116.objecter ms_dispatch 0x7f365c001550 osd_map(2..2 src has 1..2) v4 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 3 client.4116.objecter handle_osd_map decoding full epoch 2 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.286+0000 7f36537fe700 20 client.4116.objecter dump_active .. 0 homeless 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.287+0000 7f3661e3b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.287+0000 7f3661e3b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.287+0000 7f3661e3b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.287+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c141e10 con 0x7f365c0a5d10 2023-09-30T01:03:23.311 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.289+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f365c0a6190 con 0x7f365c0a5d10 2023-09-30T01:03:23.312 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.289+0000 7f36537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.312 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.289+0000 7f36537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:23.372 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.371+0000 7f3661e3b700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f8068422-b1d0-4236-a4eb-fa8aecefe228", "id": 0}] 2023-09-30T01:03:23.373 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.371+0000 7f3661e3b700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f8068422-b1d0-4236-a4eb-fa8aecefe228", "id": 0}] 2023-09-30T01:03:23.373 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.371+0000 7f3661e3b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:23.373 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.371+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f8068422-b1d0-4236-a4eb-fa8aecefe228", "id": 0} v 0) v1 -- 0x7f365c072960 con 0x7f365c0a5d10 2023-09-30T01:03:23.376 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.375+0000 7f36537fe700 1 -- 172.21.15.157:0/3887958335 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f8068422-b1d0-4236-a4eb-fa8aecefe228", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f364c01b450 con 0x7f365c0a5d10 2023-09-30T01:03:23.376 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.375+0000 7f36537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f8068422-b1d0-4236-a4eb-fa8aecefe228", "id": 0}] 2023-09-30T01:03:23.376 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.375+0000 7f36537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:23.377 INFO:teuthology.orchestra.run.smithi157.stdout:0 2023-09-30T01:03:23.378 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 10 librados: watch_flush enter 2023-09-30T01:03:23.378 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 10 librados: watch_flush exit 2023-09-30T01:03:23.378 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 20 client.4116.objecter shutdown clearing up homeless session... 2023-09-30T01:03:23.378 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 10 client.4116.objecter successfully canceled tick 2023-09-30T01:03:23.378 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 10 monclient: shutdown 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 msgr2=0x7f365c13e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 0x7f365c13e480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f364c00c940 tx=0x7f364c0016f0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 shutdown_connections 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f365c0aecc0 0x7f365c151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f365c0a5d10 0x7f365c13e480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 --2- 172.21.15.157:0/3887958335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f365c0ae8f0 0x7f365c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 >> 172.21.15.157:0/3887958335 conn(0x7f365c06a460 msgr2=0x7f365c0b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 shutdown_connections 2023-09-30T01:03:23.379 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 -- 172.21.15.157:0/3887958335 wait complete. 2023-09-30T01:03:23.380 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.376+0000 7f3661e3b700 1 librados: shutdown 2023-09-30T01:03:23.389 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new e73989ad-7372-4777-9fa6-c96c8b3bd763 1 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.505+0000 7fbe16047700 10 monclient: get_monmap_and_config 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.505+0000 7fbe16047700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.505+0000 7fbe16047700 10 monclient: monmap: 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:23.507030+0000 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:23.507030+0000 2023-09-30T01:03:23.506 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:23.507 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.506+0000 7fbe16047700 1 Processor -- start 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.506+0000 7fbe16047700 1 -- start start 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.506+0000 7fbe16047700 10 monclient: init 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T01:03:23.510 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe100ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fbe100aecc0 0x7fbe100a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-a con 0x7fbe100aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-b con 0x7fbe101415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100a7040 con 0x7fbe100ae8f0 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe100a57f0 con 0x7fbe100aecc0 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100a5480 con 0x7fbe101415c0 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.511 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe16047700 10 monclient(hunting): authenticate will time out at 1048.971213s 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0f7fe700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fbe100ae8f0 0x7fbe100b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44562/0 (socket says 172.21.15.157:44562) 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/2317504647 learned_addr learned my addr 172.21.15.157:0/2317504647 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 1 --2- 172.21.15.157:0/2317504647 >> v2:172.21.15.115:3300/0 conn(0x7fbe100aecc0 0x7fbe100a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): get_auth_request con 0x7fbe101415c0 auth_method 0 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): get_auth_request con 0x7fbe100aecc0 auth_method 0 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.512 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369628153 0 0) 0x7fbe100a7040 con 0x7fbe100ae8f0 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.513 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0e7fc700 10 monclient(hunting): my global_id is 4123 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf800bd80 con 0x7fbe100ae8f0 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 >> v2:172.21.15.115:3300/0 conn(0x7fbe100aecc0 msgr2=0x7fbe100a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> v2:172.21.15.115:3300/0 conn(0x7fbe100aecc0 0x7fbe100a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 >> v1:172.21.15.157:6789/0 conn(0x7fbe100ae8f0 legacy=0x7fbe100b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 monclient: found mon.noname-b 2023-09-30T01:03:23.514 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10141a40 con 0x7fbe101415c0 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0effd700 1 --2- 172.21.15.157:0/2317504647 >> v2:172.21.15.115:3300/0 conn(0x7fbe100aecc0 0x7fbe100a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 msgr2=0x7fbe100adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 msgr2=0x7fbe100adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fbe08009e60 tx=0x7fbe08009780 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:03:23.515 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.507+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0ffff700 10 monclient: get_auth_request con 0x7fbe101415c0 auth_method 0 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0ffff700 0 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 msgr2=0x7fbe100adb60 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe101415c0 0x7fbe100adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient: _add_conns ranks=[4,0] 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 1 --2- 172.21.15.157:0/2317504647 >> v2:172.21.15.157:3300/0 conn(0x7fbdf800c020 0x7fbdf800c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fbdf800c020 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdf800d020 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100a7040 con 0x7fbdf800d020 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf800dbf0 con 0x7fbdf800c020 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0e7fc700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 1 --2- 172.21.15.157:0/2317504647 >> v2:172.21.15.157:3300/0 conn(0x7fbdf800c020 0x7fbdf800c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): get_auth_request con 0x7fbdf800c020 auth_method 0 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.718 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.708+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_done global_id 4112 payload 293 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/2317504647 >> v1:172.21.15.115:6789/0 conn(0x7fbdf800d020 legacy=0x7fbdf800d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640918694 0 0) 0x7fbe100a7040 con 0x7fbdf800d020 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 1 monclient: found mon.noname-c 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.719 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/2317504647 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf800de20 con 0x7fbdf800c020 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0f7fe700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbdf800c420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800cfa0 tx=0x7fbe00007de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330529ea63f423d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00005a40 con 0x7fbdf800c020 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: dump: 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.720 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe16047700 5 monclient: authenticate success, global_id 4112 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe16047700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00005ba0 con 0x7fbdf800c020 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe16047700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:23.721 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 1 -- 172.21.15.157:0/2317504647 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000070d0 con 0x7fbdf800c020 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe16047700 4 set_mon_vals no callback set 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.709+0000 7fbe0e7fc700 10 monclient: dump: 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.722 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 10 monclient: shutdown 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 -- 172.21.15.157:0/2317504647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 msgr2=0x7fbdf800c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbdf800c420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800cfa0 tx=0x7fbe00007de0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe0ffff700 1 -- 172.21.15.157:0/2317504647 reap_dead start 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 -- 172.21.15.157:0/2317504647 shutdown_connections 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 --2- 172.21.15.157:0/2317504647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbdf800c420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 -- 172.21.15.157:0/2317504647 >> 172.21.15.157:0/2317504647 conn(0x7fbe1006a460 msgr2=0x7fbe100630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 -- 172.21.15.157:0/2317504647 shutdown_connections 2023-09-30T01:03:23.723 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 1 -- 172.21.15.157:0/2317504647 wait complete. 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.710+0000 7fbe16047700 10 monclient: monmap: 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.724 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 librados: starting msgr at 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 librados: starting objecter 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 Processor -- start 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 -- start start 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 librados: setting wanted keys 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 librados: calling monclient init 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 10 monclient: init 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-b con 0x7fbdf800c020 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe100a83f0 0x7fbe100a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.725 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.711+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-c con 0x7fbe100a83f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe100a95b0 0x7fbe1014cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): picked mon.noname-a con 0x7fbe100a95b0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100a78d0 con 0x7fbe100a95b0 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100a8b50 con 0x7fbdf800c020 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100a8860 con 0x7fbe100a83f0 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient(hunting): authenticate will time out at 1049.176036s 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.726 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57310/0 (socket says 172.21.15.157:57310) 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/1946580622 learned_addr learned my addr 172.21.15.157:0/1946580622 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): get_auth_request con 0x7fbdf800c020 auth_method 0 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe100a83f0 0x7fbe100a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): get_auth_request con 0x7fbe100a83f0 auth_method 0 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.727 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe100a95b0 0x7fbe1014cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): get_auth_request con 0x7fbe100a95b0 auth_method 0 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.728 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe100a83f0 msgr2=0x7fbe100a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe100a83f0 0x7fbe100a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe100a95b0 msgr2=0x7fbe1014cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe100a95b0 0x7fbe1014cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 monclient: found mon.noname-b 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 -- 172.21.15.157:0/1946580622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1014a540 con 0x7fbdf800c020 2023-09-30T01:03:23.729 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0f7fe700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe00001ed0 tx=0x7fbe00015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 client.?.objecter ms_handle_connect 0x7fbdf800c020 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000070d0 con 0x7fbdf800c020 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: dump: 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.730 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe000014a0 con 0x7fbdf800c020 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 5 monclient: authenticate success, global_id 4126 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000c3f0 con 0x7fbdf800c020 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient: _renew_subs 2023-09-30T01:03:23.731 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1014a6d0 con 0x7fbdf800c020 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 10 monclient: dump: 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.732 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe0dffb700 4 set_mon_vals no callback set 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient: _renew_subs 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1006d0b0 con 0x7fbdf800c020 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbe16047700 1 librados: init done 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbe00019030 con 0x7fbdf800c020 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.712+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fbe00005190 con 0x7fbdf800c020 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbdf77fe700 10 client.4126.objecter ms_dispatch 0x7fbe10001550 osd_map(3..3 src has 1..3) v4 2023-09-30T01:03:23.733 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbdf77fe700 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbdf77fe700 3 client.4126.objecter handle_osd_map decoding full epoch 3 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbdf77fe700 20 client.4126.objecter dump_active .. 0 homeless 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbe16047700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbe16047700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbe16047700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.713+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe100adba0 con 0x7fbdf800c020 2023-09-30T01:03:23.734 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.717+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbe000466d0 con 0x7fbdf800c020 2023-09-30T01:03:23.735 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.717+0000 7fbdf77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.735 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.717+0000 7fbdf77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:23.749 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:23.749+0000 7fa3f6d3d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T01:03:23.787 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:23.786+0000 7fa6f95401c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T01:03:23.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.801+0000 7fbe16047700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e73989ad-7372-4777-9fa6-c96c8b3bd763", "id": 1}] 2023-09-30T01:03:23.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.801+0000 7fbe16047700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e73989ad-7372-4777-9fa6-c96c8b3bd763", "id": 1}] 2023-09-30T01:03:23.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.801+0000 7fbe16047700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.801+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e73989ad-7372-4777-9fa6-c96c8b3bd763", "id": 1} v 0) v1 -- 0x7fbe10072960 con 0x7fbdf800c020 2023-09-30T01:03:23.804 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbdf77fe700 1 -- 172.21.15.157:0/1946580622 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e73989ad-7372-4777-9fa6-c96c8b3bd763", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbe00035030 con 0x7fbdf800c020 2023-09-30T01:03:23.804 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbdf77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e73989ad-7372-4777-9fa6-c96c8b3bd763", "id": 1}] 2023-09-30T01:03:23.805 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbdf77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:23.805 INFO:teuthology.orchestra.run.smithi157.stdout:1 2023-09-30T01:03:23.806 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbe16047700 10 librados: watch_flush enter 2023-09-30T01:03:23.806 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbe16047700 10 librados: watch_flush exit 2023-09-30T01:03:23.806 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbe16047700 20 client.4126.objecter shutdown clearing up homeless session... 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.804+0000 7fbe16047700 10 client.4126.objecter successfully canceled tick 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 10 monclient: shutdown 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 msgr2=0x7fbe10149aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe00001ed0 tx=0x7fbe00015000 comp rx=0 tx=0).stop 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 shutdown_connections 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe100a95b0 0x7fbe1014cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdf800c020 0x7fbe10149aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 --2- 172.21.15.157:0/1946580622 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fbe100a83f0 0x7fbe100a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 >> 172.21.15.157:0/1946580622 conn(0x7fbe1006a460 msgr2=0x7fbe100b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 shutdown_connections 2023-09-30T01:03:23.807 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 -- 172.21.15.157:0/1946580622 wait complete. 2023-09-30T01:03:23.808 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.805+0000 7fbe16047700 1 librados: shutdown 2023-09-30T01:03:23.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new d8599e95-3db1-402a-a597-42d388a61a64 2 2023-09-30T01:03:23.885 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:23.885+0000 7fa3f6d3d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T01:03:23.935 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:23.934+0000 7fa6f95401c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.934+0000 7fd4ca26c700 10 monclient: get_monmap_and_config 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.934+0000 7fd4ca26c700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.934+0000 7fd4ca26c700 10 monclient: monmap: 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:23.936398+0000 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:23.936398+0000 2023-09-30T01:03:23.936 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.935+0000 7fd4ca26c700 1 Processor -- start 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.935+0000 7fd4ca26c700 1 -- start start 2023-09-30T01:03:23.937 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.935+0000 7fd4ca26c700 10 monclient: init 2023-09-30T01:03:23.946 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.946 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-09-30T01:03:23.946 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4c40ae8f0 0x7fd4c40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.946 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-c con 0x7fd4c40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4c40aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4c41415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40a7040 con 0x7fd4c41415c0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40a57f0 con 0x7fd4c40aecc0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c40a5480 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient(hunting): authenticate will time out at 1049.400115s 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4c40ae8f0 0x7fd4c40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.947 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c37fe700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd4c40aecc0 0x7fd4c40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44576/0 (socket says 172.21.15.157:44576) 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c37fe700 1 -- 172.21.15.157:0/2620779295 learned_addr learned my addr 172.21.15.157:0/2620779295 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): get_auth_request con 0x7fd4c40ae8f0 auth_method 0 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-09-30T01:03:23.948 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 -- 172.21.15.157:0/2620779295 >> v1:172.21.15.157:6789/0 conn(0x7fd4c40aecc0 legacy=0x7fd4c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 -- 172.21.15.157:0/2620779295 >> v1:172.21.15.115:6789/0 conn(0x7fd4c41415c0 legacy=0x7fd4c40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 monclient: found mon.noname-c 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 -- 172.21.15.157:0/2620779295 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40a55c0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c3fff700 1 --2- 172.21.15.157:0/2620779295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c40a5d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40098e0 tx=0x7fd4b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ebcf619948ed2b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 1 -- 172.21.15.157:0/2620779295 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4009e80 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:23.949 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: dump: 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.950 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 1 -- 172.21.15.157:0/2620779295 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b400b860 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 5 monclient: authenticate success, global_id 4129 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 1 -- 172.21.15.157:0/2620779295 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b400bbe0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4ca26c700 4 set_mon_vals no callback set 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.936+0000 7fd4c2ffd700 10 monclient: dump: 2023-09-30T01:03:23.951 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.952 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 10 monclient: shutdown 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 -- 172.21.15.157:0/2620779295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 msgr2=0x7fd4c40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/2620779295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c40a5d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40098e0 tx=0x7fd4b40091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 -- 172.21.15.157:0/2620779295 shutdown_connections 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/2620779295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c40a5d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 -- 172.21.15.157:0/2620779295 >> 172.21.15.157:0/2620779295 conn(0x7fd4c406a460 msgr2=0x7fd4c40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 -- 172.21.15.157:0/2620779295 shutdown_connections 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.937+0000 7fd4ca26c700 1 -- 172.21.15.157:0/2620779295 wait complete. 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 10 monclient: build_initial_monmap 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 10 monclient: monmap: 2023-09-30T01:03:23.953 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 librados: starting msgr at 2023-09-30T01:03:23.954 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 librados: starting objecter 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 Processor -- start 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 -- start start 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 librados: setting wanted keys 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 1 librados: calling monclient init 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.938+0000 7fd4ca26c700 10 monclient: init 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-b con 0x7fd4c40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd4c40aecc0 0x7fd4c414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-c con 0x7fd4c40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:23.955 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c41415c0 0x7fd4c40a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): picked mon.noname-a con 0x7fd4c41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c414e1a0 con 0x7fd4c41415c0 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c414dec0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c414eb70 con 0x7fd4c40aecc0 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4ca26c700 10 monclient(hunting): authenticate will time out at 1049.403277s 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57318/0 (socket says 172.21.15.157:57318) 2023-09-30T01:03:23.956 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 -- 172.21.15.157:0/191848444 learned_addr learned my addr 172.21.15.157:0/191848444 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): get_auth_request con 0x7fd4c40ae8f0 auth_method 0 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c41415c0 0x7fd4c40a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd4c40aecc0 0x7fd4c414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): get_auth_request con 0x7fd4c41415c0 auth_method 0 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): get_auth_request con 0x7fd4c40aecc0 auth_method 0 2023-09-30T01:03:23.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c8809700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:23.958 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 -- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd4c40aecc0 msgr2=0x7fd4c414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd4c40aecc0 0x7fd4c414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 -- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c41415c0 msgr2=0x7fd4c40a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c41415c0 0x7fd4c40a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 monclient: found mon.noname-b 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 -- 172.21.15.157:0/191848444 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b4009420 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4c3fff700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40b0920 tx=0x7fd4b4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:23.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4abfff700 10 client.?.objecter ms_handle_connect 0x7fd4c40ae8f0 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.939+0000 7fd4abfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b400b9d0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: dump: 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.960 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 5 monclient: authenticate success, global_id 4135 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b400bb30 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 10 monclient: _renew_subs 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c414edf0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:23.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4025650 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 monclient: dump: 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:23.962 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 10 monclient: _renew_subs 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4c27fc700 4 set_mon_vals no callback set 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c40a9cf0 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4ca26c700 1 librados: init done 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd4b4023070 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fd4b4011850 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 10 client.4135.objecter ms_dispatch 0x7fd4c4001550 osd_map(4..4 src has 1..4) v4 2023-09-30T01:03:23.963 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 3 client.4135.objecter handle_osd_map got epochs [4,4] > 0 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 3 client.4135.objecter handle_osd_map decoding full epoch 4 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.940+0000 7fd4abfff700 20 client.4135.objecter dump_active .. 0 homeless 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.941+0000 7fd4ca26c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.941+0000 7fd4ca26c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.941+0000 7fd4ca26c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.941+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c414e480 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.943+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd4b402a070 con 0x7fd4c40ae8f0 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.943+0000 7fd4abfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:23.964 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:23.943+0000 7fd4abfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:24.026 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.025+0000 7fd4ca26c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8599e95-3db1-402a-a597-42d388a61a64", "id": 2}] 2023-09-30T01:03:24.026 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.025+0000 7fd4ca26c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8599e95-3db1-402a-a597-42d388a61a64", "id": 2}] 2023-09-30T01:03:24.026 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.025+0000 7fd4ca26c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.026 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.025+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d8599e95-3db1-402a-a597-42d388a61a64", "id": 2} v 0) v1 -- 0x7fd4c4072960 con 0x7fd4c40ae8f0 2023-09-30T01:03:24.028 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.028+0000 7fd4abfff700 1 -- 172.21.15.157:0/191848444 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8599e95-3db1-402a-a597-42d388a61a64", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd4b4015020 con 0x7fd4c40ae8f0 2023-09-30T01:03:24.029 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.028+0000 7fd4abfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8599e95-3db1-402a-a597-42d388a61a64", "id": 2}] 2023-09-30T01:03:24.029 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.028+0000 7fd4abfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:24.029 INFO:teuthology.orchestra.run.smithi157.stdout:2 2023-09-30T01:03:24.030 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 10 librados: watch_flush enter 2023-09-30T01:03:24.030 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 10 librados: watch_flush exit 2023-09-30T01:03:24.030 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 20 client.4135.objecter shutdown clearing up homeless session... 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 10 client.4135.objecter successfully canceled tick 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 10 monclient: shutdown 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 msgr2=0x7fd4c413d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40b0920 tx=0x7fd4b4017000 comp rx=0 tx=0).stop 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 shutdown_connections 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd4c41415c0 0x7fd4c40a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c40ae8f0 0x7fd4c413d5c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 --2- 172.21.15.157:0/191848444 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd4c40aecc0 0x7fd4c414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 >> 172.21.15.157:0/191848444 conn(0x7fd4c406a460 msgr2=0x7fd4c413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.031 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 shutdown_connections 2023-09-30T01:03:24.032 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 -- 172.21.15.157:0/191848444 wait complete. 2023-09-30T01:03:24.032 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.029+0000 7fd4ca26c700 1 librados: shutdown 2023-09-30T01:03:24.041 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 08039bd8-9791-4ea6-a7d6-638fa734cdaf 3 2023-09-30T01:03:24.159 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.158+0000 7f6c5c0d1700 10 monclient: get_monmap_and_config 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.158+0000 7f6c5c0d1700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.158+0000 7f6c5c0d1700 10 monclient: monmap: 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:24.160424+0000 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:24.160424+0000 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.160 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.161 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.159+0000 7f6c5c0d1700 1 Processor -- start 2023-09-30T01:03:24.171 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.159+0000 7f6c5c0d1700 1 -- start start 2023-09-30T01:03:24.171 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.159+0000 7f6c5c0d1700 10 monclient: init 2023-09-30T01:03:24.171 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.171 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c540ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 0x7f6c540a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-b con 0x7f6c540aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c541415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540a7040 con 0x7f6c541415c0 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540a57f0 con 0x7f6c540ae8f0 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540a5480 con 0x7f6c540aecc0 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5c0d1700 10 monclient(hunting): authenticate will time out at 1049.624135s 2023-09-30T01:03:24.172 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c59e6d700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6c540ae8f0 0x7f6c540b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44580/0 (socket says 172.21.15.157:44580) 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c59e6d700 1 -- 172.21.15.157:0/24080302 learned_addr learned my addr 172.21.15.157:0/24080302 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 --2- 172.21.15.157:0/24080302 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 0x7f6c540a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): get_auth_request con 0x7f6c540aecc0 auth_method 0 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595295363 0 0) 0x7f6c540a57f0 con 0x7f6c540ae8f0 2023-09-30T01:03:24.173 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): my global_id is 4138 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c40003570 con 0x7f6c540ae8f0 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609420883 0 0) 0x7f6c540a7040 con 0x7f6c541415c0 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient(hunting): my global_id is 4131 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c540a57f0 con 0x7f6c541415c0 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2023-09-30T01:03:24.174 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 -- 172.21.15.157:0/24080302 >> v1:172.21.15.157:6789/0 conn(0x7f6c540ae8f0 legacy=0x7f6c540b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 -- 172.21.15.157:0/24080302 >> v1:172.21.15.115:6789/0 conn(0x7f6c541415c0 legacy=0x7f6c540adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 monclient: found mon.noname-b 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 -- 172.21.15.157:0/24080302 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540a55c0 con 0x7f6c540aecc0 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c5966c700 1 --2- 172.21.15.157:0/24080302 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 0x7f6c540a5d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009910 tx=0x7f6c440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c443677c3973b7a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44011630 con 0x7f6c540aecc0 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.175 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.160+0000 7f6c58e6b700 10 monclient: dump: 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.176 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 5 monclient: authenticate success, global_id 4121 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44011790 con 0x7f6c540aecc0 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 1 -- 172.21.15.157:0/24080302 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44011b30 con 0x7f6c540aecc0 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 4 set_mon_vals no callback set 2023-09-30T01:03:24.177 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c58e6b700 10 monclient: dump: 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.178 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 10 monclient: shutdown 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/24080302 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 msgr2=0x7f6c540a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/24080302 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 0x7f6c540a5d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009910 tx=0x7f6c440091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/24080302 shutdown_connections 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/24080302 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c540aecc0 0x7f6c540a5d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/24080302 >> 172.21.15.157:0/24080302 conn(0x7f6c5406a460 msgr2=0x7f6c540630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/24080302 shutdown_connections 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.161+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/24080302 wait complete. 2023-09-30T01:03:24.179 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 10 monclient: monmap: 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:24.180 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 librados: starting msgr at 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 librados: starting objecter 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 Processor -- start 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 -- start start 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 librados: setting wanted keys 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 1 librados: calling monclient init 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.162+0000 7f6c5c0d1700 10 monclient: init 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.181 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-b con 0x7f6c540ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c540aecc0 0x7f6c54151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-a con 0x7f6c540aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c541415c0 0x7f6c541518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): picked mon.noname-c con 0x7f6c541415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540a9920 con 0x7f6c540aecc0 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540a7830 con 0x7f6c540ae8f0 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540a79a0 con 0x7f6c541415c0 2023-09-30T01:03:24.182 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 10 monclient(hunting): authenticate will time out at 1049.627273s 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57320/0 (socket says 172.21.15.157:57320) 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 -- 172.21.15.157:0/1659552156 learned_addr learned my addr 172.21.15.157:0/1659552156 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): get_auth_request con 0x7f6c540ae8f0 auth_method 0 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c540aecc0 0x7f6c54151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): get_auth_request con 0x7f6c540aecc0 auth_method 0 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c541415c0 0x7f6c541518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.183 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): get_auth_request con 0x7f6c541415c0 auth_method 0 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.184 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5966c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5a66e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 -- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c541415c0 msgr2=0x7f6c541518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c541415c0 0x7f6c541518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 -- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c540aecc0 msgr2=0x7f6c54151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c540aecc0 0x7f6c54151380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 monclient: found mon.noname-b 2023-09-30T01:03:24.185 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 -- 172.21.15.157:0/1659552156 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c44009420 con 0x7f6c540ae8f0 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c59e6d700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000c790 tx=0x7f6c50007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 client.?.objecter ms_handle_connect 0x7f6c540ae8f0 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c50005b30 con 0x7f6c540ae8f0 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 monclient: dump: 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.186 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c5c0d1700 5 monclient: authenticate success, global_id 4141 2023-09-30T01:03:24.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c50005cd0 con 0x7f6c540ae8f0 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.163+0000 7f6c49ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 10 monclient: _renew_subs 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c50007320 con 0x7f6c540ae8f0 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c54151e00 con 0x7f6c540ae8f0 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c4bfff700 4 set_mon_vals no callback set 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 10 monclient: dump: 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.188 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6c50012680 con 0x7f6c540ae8f0 2023-09-30T01:03:24.189 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 10 monclient: _renew_subs 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c540a96d0 con 0x7f6c540ae8f0 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c5c0d1700 1 librados: init done 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f6c50005e40 con 0x7f6c540ae8f0 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 10 client.4141.objecter ms_dispatch 0x7f6c54001550 osd_map(5..5 src has 1..5) v4 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 3 client.4141.objecter handle_osd_map got epochs [5,5] > 0 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 3 client.4141.objecter handle_osd_map decoding full epoch 5 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.164+0000 7f6c49ffb700 20 client.4141.objecter dump_active .. 0 homeless 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.165+0000 7f6c5c0d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.165+0000 7f6c5c0d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.165+0000 7f6c5c0d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.165+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c54072960 con 0x7f6c540ae8f0 2023-09-30T01:03:24.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.167+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6c50017070 con 0x7f6c540ae8f0 2023-09-30T01:03:24.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.167+0000 7f6c49ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.167+0000 7f6c49ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:24.220 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:24.220+0000 7fa3f6d3d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:03:24.250 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.249+0000 7f6c5c0d1700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "08039bd8-9791-4ea6-a7d6-638fa734cdaf", "id": 3}] 2023-09-30T01:03:24.250 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.249+0000 7f6c5c0d1700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "08039bd8-9791-4ea6-a7d6-638fa734cdaf", "id": 3}] 2023-09-30T01:03:24.250 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.249+0000 7f6c5c0d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.250 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.249+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "08039bd8-9791-4ea6-a7d6-638fa734cdaf", "id": 3} v 0) v1 -- 0x7f6c54141a40 con 0x7f6c540ae8f0 2023-09-30T01:03:24.253 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.252+0000 7f6c49ffb700 1 -- 172.21.15.157:0/1659552156 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08039bd8-9791-4ea6-a7d6-638fa734cdaf", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6c50007710 con 0x7f6c540ae8f0 2023-09-30T01:03:24.253 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.252+0000 7f6c49ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "08039bd8-9791-4ea6-a7d6-638fa734cdaf", "id": 3}] 2023-09-30T01:03:24.253 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.252+0000 7f6c49ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:24.253 INFO:teuthology.orchestra.run.smithi157.stdout:3 2023-09-30T01:03:24.254 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 10 librados: watch_flush enter 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 10 librados: watch_flush exit 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 20 client.4141.objecter shutdown clearing up homeless session... 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 10 client.4141.objecter successfully canceled tick 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 10 monclient: shutdown 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 msgr2=0x7f6c540a86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000c790 tx=0x7f6c50007e80 comp rx=0 tx=0).stop 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 shutdown_connections 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6c541415c0 0x7f6c541518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c540ae8f0 0x7f6c540a86a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.255 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 --2- 172.21.15.157:0/1659552156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c540aecc0 0x7f6c54151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.256 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 >> 172.21.15.157:0/1659552156 conn(0x7f6c5406a460 msgr2=0x7f6c5413ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.256 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 shutdown_connections 2023-09-30T01:03:24.256 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 -- 172.21.15.157:0/1659552156 wait complete. 2023-09-30T01:03:24.256 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.253+0000 7f6c5c0d1700 1 librados: shutdown 2023-09-30T01:03:24.267 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 38c57cc5-98c8-4d46-913a-a119cf08ed58 4 2023-09-30T01:03:24.294 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:24.293+0000 7fa6f95401c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.386+0000 7fe30c196700 10 monclient: get_monmap_and_config 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.386+0000 7fe30c196700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.387+0000 7fe30c196700 10 monclient: monmap: 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.401 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:24.388713+0000 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:24.388713+0000 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.402 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.387+0000 7fe30c196700 1 Processor -- start 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.387+0000 7fe30c196700 1 -- start start 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.387+0000 7fe30c196700 10 monclient: init 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 0x7fe3040a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-b con 0x7fe3040ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3040aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe3041415c0 0x7fe3040adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-a con 0x7fe3041415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3040a7040 con 0x7fe3040aecc0 2023-09-30T01:03:24.403 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3040a57f0 con 0x7fe3041415c0 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040a5480 con 0x7fe3040ae8f0 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30c196700 10 monclient(hunting): authenticate will time out at 1049.852450s 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309731700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe3040aecc0 0x7fe3040b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.157:46696/0 (socket says 172.21.15.157:46696) 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe3041415c0 0x7fe3040adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309731700 1 -- 172.21.15.157:0/781946580 learned_addr learned my addr 172.21.15.157:0/781946580 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 0x7fe3040a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.404 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): get_auth_request con 0x7fe3041415c0 auth_method 0 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): get_auth_request con 0x7fe3040ae8f0 auth_method 0 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.405 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 1 -- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 msgr2=0x7fe3040a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 0x7fe3040a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe30a733700 1 -- 172.21.15.157:0/781946580 >> v1:172.21.15.115:6789/0 conn(0x7fe3040aecc0 legacy=0x7fe3040b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 1 monclient: found mon.noname-a 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.406 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 1 -- 172.21.15.157:0/781946580 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3040a55c0 con 0x7fe3041415c0 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.388+0000 7fe308f30700 1 -- 172.21.15.157:0/781946580 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447976468 0 0) 0x7fe3040a7040 con 0x7fe3040aecc0 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe309f32700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 0x7fe3040a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30a733700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3041415c0 0x7fe3040adb60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009e60 tx=0x7fe2fc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797f919be664aff7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 1 -- 172.21.15.157:0/781946580 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc00e070 con 0x7fe3041415c0 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: dump: 2023-09-30T01:03:24.407 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.408 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 1 -- 172.21.15.157:0/781946580 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc00bb60 con 0x7fe3041415c0 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 5 monclient: authenticate success, global_id 4137 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 1 -- 172.21.15.157:0/781946580 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc012430 con 0x7fe3041415c0 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 4 set_mon_vals no callback set 2023-09-30T01:03:24.409 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe308f30700 10 monclient: dump: 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.410 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 10 monclient: shutdown 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 -- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3041415c0 msgr2=0x7fe3040adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3041415c0 0x7fe3040adb60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009e60 tx=0x7fe2fc009780 comp rx=0 tx=0).stop 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 -- 172.21.15.157:0/781946580 shutdown_connections 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3040ae8f0 0x7fe3040a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 --2- 172.21.15.157:0/781946580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3041415c0 0x7fe3040adb60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 -- 172.21.15.157:0/781946580 >> 172.21.15.157:0/781946580 conn(0x7fe30406a460 msgr2=0x7fe3040630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.411 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.389+0000 7fe30c196700 1 -- 172.21.15.157:0/781946580 shutdown_connections 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 -- 172.21.15.157:0/781946580 wait complete. 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 10 monclient: monmap: 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:24.412 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 librados: starting msgr at 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 librados: starting objecter 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 Processor -- start 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 -- start start 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 librados: setting wanted keys 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 1 librados: calling monclient init 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.390+0000 7fe30c196700 10 monclient: init 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.413 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3040ae8f0 0x7fe30413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-a con 0x7fe3040ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-b con 0x7fe3040aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 0x7fe3040a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): picked mon.noname-c con 0x7fe3041415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040a8730 con 0x7fe3040ae8f0 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040ab5f0 con 0x7fe3040aecc0 2023-09-30T01:03:24.414 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040a8170 con 0x7fe3041415c0 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30c196700 10 monclient(hunting): authenticate will time out at 1049.855411s 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57332/0 (socket says 172.21.15.157:57332) 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 -- 172.21.15.157:0/4145853731 learned_addr learned my addr 172.21.15.157:0/4145853731 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 0x7fe3040a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): get_auth_request con 0x7fe3040aecc0 auth_method 0 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.415 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3040ae8f0 0x7fe30413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): get_auth_request con 0x7fe3041415c0 auth_method 0 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): get_auth_request con 0x7fe3040ae8f0 auth_method 0 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.416 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309f32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-09-30T01:03:24.417 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 -- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 msgr2=0x7fe3040a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 0x7fe3040a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 -- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3040ae8f0 msgr2=0x7fe30413e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3040ae8f0 0x7fe30413e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 monclient: found mon.noname-b 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe309731700 1 -- 172.21.15.157:0/4145853731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300009420 con 0x7fe3040aecc0 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.391+0000 7fe30a733700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 0x7fe3040a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:24.418 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe309731700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400c7e0 tx=0x7fe2f4007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 client.?.objecter ms_handle_connect 0x7fe3040aecc0 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4005b30 con 0x7fe3040aecc0 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: dump: 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.419 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4005c90 con 0x7fe3040aecc0 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 5 monclient: authenticate success, global_id 4144 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4007300 con 0x7fe3040aecc0 2023-09-30T01:03:24.420 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 10 monclient: _renew_subs 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc009a20 con 0x7fe3040aecc0 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 monclient: dump: 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.421 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2fbfff700 4 set_mon_vals no callback set 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe2f4012680 con 0x7fe3040aecc0 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 10 monclient: _renew_subs 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040a9cd0 con 0x7fe3040aecc0 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe30c196700 1 librados: init done 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fe2f4005e00 con 0x7fe3040aecc0 2023-09-30T01:03:24.422 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 10 client.4144.objecter ms_dispatch 0x7fe304001550 osd_map(6..6 src has 1..6) v4 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 3 client.4144.objecter handle_osd_map decoding full epoch 6 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.392+0000 7fe2f9ffb700 20 client.4144.objecter dump_active .. 0 homeless 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.393+0000 7fe30c196700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.393+0000 7fe30c196700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.393+0000 7fe30c196700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.393+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3040a9270 con 0x7fe3040aecc0 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.395+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe3040a9270 con 0x7fe3040aecc0 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.395+0000 7fe2f9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.423 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.395+0000 7fe2f9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:24.479 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.478+0000 7fe30c196700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "38c57cc5-98c8-4d46-913a-a119cf08ed58", "id": 4}] 2023-09-30T01:03:24.479 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.478+0000 7fe30c196700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "38c57cc5-98c8-4d46-913a-a119cf08ed58", "id": 4}] 2023-09-30T01:03:24.479 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.478+0000 7fe30c196700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.480 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.478+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "38c57cc5-98c8-4d46-913a-a119cf08ed58", "id": 4} v 0) v1 -- 0x7fe304072960 con 0x7fe3040aecc0 2023-09-30T01:03:24.481 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.481+0000 7fe2f9ffb700 1 -- 172.21.15.157:0/4145853731 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38c57cc5-98c8-4d46-913a-a119cf08ed58", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe2f4017070 con 0x7fe3040aecc0 2023-09-30T01:03:24.482 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.481+0000 7fe2f9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "38c57cc5-98c8-4d46-913a-a119cf08ed58", "id": 4}] 2023-09-30T01:03:24.482 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.481+0000 7fe2f9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:24.482 INFO:teuthology.orchestra.run.smithi157.stdout:4 2023-09-30T01:03:24.484 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 10 librados: watch_flush enter 2023-09-30T01:03:24.484 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 10 librados: watch_flush exit 2023-09-30T01:03:24.484 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 20 client.4144.objecter shutdown clearing up homeless session... 2023-09-30T01:03:24.484 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 10 client.4144.objecter successfully canceled tick 2023-09-30T01:03:24.484 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 10 monclient: shutdown 2023-09-30T01:03:24.485 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.485 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 msgr2=0x7fe3040a7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.485 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400c7e0 tx=0x7fe2f4007df0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.485 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 shutdown_connections 2023-09-30T01:03:24.485 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe3041415c0 0x7fe3040a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3040ae8f0 0x7fe30413e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 --2- 172.21.15.157:0/4145853731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3040aecc0 0x7fe3040a7c30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 >> 172.21.15.157:0/4145853731 conn(0x7fe30406a460 msgr2=0x7fe3040b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 shutdown_connections 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 -- 172.21.15.157:0/4145853731 wait complete. 2023-09-30T01:03:24.486 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.482+0000 7fe30c196700 1 librados: shutdown 2023-09-30T01:03:24.494 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new fae1ca7c-8b05-406f-bfa7-c61093a32111 5 2023-09-30T01:03:24.651 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.650+0000 7fd973814700 10 monclient: get_monmap_and_config 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.650+0000 7fd973814700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.650+0000 7fd973814700 10 monclient: monmap: 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:24.652456+0000 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:24.652456+0000 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:24.652 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:24.653 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:24.653 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:24.653 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.653 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.651+0000 7fd973814700 1 Processor -- start 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.651+0000 7fd973814700 1 -- start start 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.651+0000 7fd973814700 10 monclient: init 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd96c0ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:24.663 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 0x7fd96c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): picked mon.noname-b con 0x7fd96c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd96c1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c0a7040 con 0x7fd96c1415c0 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c0a57f0 con 0x7fd96c0ae8f0 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0a5480 con 0x7fd96c0aecc0 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd973814700 10 monclient(hunting): authenticate will time out at 1050.116173s 2023-09-30T01:03:24.664 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd9715b0700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd96c0ae8f0 0x7fd96c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44584/0 (socket says 172.21.15.157:44584) 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd9715b0700 1 -- 172.21.15.157:0/1553419260 learned_addr learned my addr 172.21.15.157:0/1553419260 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 --2- 172.21.15.157:0/1553419260 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 0x7fd96c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): get_auth_request con 0x7fd96c0aecc0 auth_method 0 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381893616 0 0) 0x7fd96c0a57f0 con 0x7fd96c0ae8f0 2023-09-30T01:03:24.665 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): my global_id is 4147 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd958003570 con 0x7fd96c0ae8f0 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701369849 0 0) 0x7fd96c0a7040 con 0x7fd96c1415c0 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 10 monclient(hunting): my global_id is 4146 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c0a57f0 con 0x7fd96c1415c0 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient(hunting): handle_auth_done global_id 4133 payload 274 2023-09-30T01:03:24.666 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 -- 172.21.15.157:0/1553419260 >> v1:172.21.15.157:6789/0 conn(0x7fd96c0ae8f0 legacy=0x7fd96c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 -- 172.21.15.157:0/1553419260 >> v1:172.21.15.115:6789/0 conn(0x7fd96c1415c0 legacy=0x7fd96c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 monclient: found mon.noname-b 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 -- 172.21.15.157:0/1553419260 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0a55c0 con 0x7fd96c0aecc0 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.652+0000 7fd970daf700 1 --2- 172.21.15.157:0/1553419260 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 0x7fd96c0a5d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009910 tx=0x7fd95c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a35cfa65cd1ff3b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c011630 con 0x7fd96c0aecc0 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.667 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: dump: 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.668 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.670 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 5 monclient: authenticate success, global_id 4133 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c011790 con 0x7fd96c0aecc0 2023-09-30T01:03:24.671 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 4 set_mon_vals no callback set 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 1 -- 172.21.15.157:0/1553419260 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c011b30 con 0x7fd96c0aecc0 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd963fff700 10 monclient: dump: 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.672 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 10 monclient: shutdown 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 -- 172.21.15.157:0/1553419260 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 msgr2=0x7fd96c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 --2- 172.21.15.157:0/1553419260 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 0x7fd96c0a5d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009910 tx=0x7fd95c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.673 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 -- 172.21.15.157:0/1553419260 shutdown_connections 2023-09-30T01:03:24.674 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 --2- 172.21.15.157:0/1553419260 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0aecc0 0x7fd96c0a5d10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.674 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 -- 172.21.15.157:0/1553419260 >> 172.21.15.157:0/1553419260 conn(0x7fd96c06a460 msgr2=0x7fd96c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.674 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 -- 172.21.15.157:0/1553419260 shutdown_connections 2023-09-30T01:03:24.674 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.653+0000 7fd973814700 1 -- 172.21.15.157:0/1553419260 wait complete. 2023-09-30T01:03:24.674 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 10 monclient: monmap: 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:24.675 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 librados: starting msgr at 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 librados: starting objecter 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 Processor -- start 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 -- start start 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 librados: setting wanted keys 2023-09-30T01:03:24.676 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 1 librados: calling monclient init 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.654+0000 7fd973814700 10 monclient: init 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0ae8f0 0x7fd96c0a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): picked mon.noname-c con 0x7fd96c0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): picked mon.noname-b con 0x7fd96c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:24.677 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd96c1415c0 0x7fd96c0a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): picked mon.noname-a con 0x7fd96c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c13e2e0 con 0x7fd96c1415c0 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0ab500 con 0x7fd96c0aecc0 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c13ec20 con 0x7fd96c0ae8f0 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.678 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd973814700 10 monclient(hunting): authenticate will time out at 1050.119463s 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57348/0 (socket says 172.21.15.157:57348) 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 -- 172.21.15.157:0/3604000442 learned_addr learned my addr 172.21.15.157:0/3604000442 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): get_auth_request con 0x7fd96c0aecc0 auth_method 0 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.679 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.680 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0ae8f0 0x7fd96c0a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.680 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): get_auth_request con 0x7fd96c0ae8f0 auth_method 0 2023-09-30T01:03:24.680 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd96c1415c0 0x7fd96c0a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): get_auth_request con 0x7fd96c1415c0 auth_method 0 2023-09-30T01:03:24.681 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd9715b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd971db1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.682 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 -- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0ae8f0 msgr2=0x7fd96c0a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0ae8f0 0x7fd96c0a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 -- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd96c1415c0 msgr2=0x7fd96c0a9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd96c1415c0 0x7fd96c0a9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 monclient: found mon.noname-b 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.655+0000 7fd970daf700 1 -- 172.21.15.157:0/3604000442 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c009420 con 0x7fd96c0aecc0 2023-09-30T01:03:24.683 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd970daf700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009fd0 tx=0x7fd95c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 client.?.objecter ms_handle_connect 0x7fd96c0aecc0 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c01f070 con 0x7fd96c0aecc0 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: dump: 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.684 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 5 monclient: authenticate success, global_id 4150 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c002f70 con 0x7fd96c0aecc0 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.685 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c011900 con 0x7fd96c0aecc0 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 10 monclient: _renew_subs 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c0aa050 con 0x7fd96c0aecc0 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9637fe700 4 set_mon_vals no callback set 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.686 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 monclient: dump: 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.687 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 10 monclient: _renew_subs 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c151020 con 0x7fd96c0aecc0 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd973814700 1 librados: init done 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd95c023070 con 0x7fd96c0aecc0 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fd95c011a80 con 0x7fd96c0aecc0 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 10 client.4150.objecter ms_dispatch 0x7fd96c001550 osd_map(7..7 src has 1..7) v4 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 3 client.4150.objecter handle_osd_map decoding full epoch 7 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.656+0000 7fd9617fa700 20 client.4150.objecter dump_active .. 0 homeless 2023-09-30T01:03:24.688 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.657+0000 7fd973814700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.657+0000 7fd973814700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.657+0000 7fd973814700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.657+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c0aa320 con 0x7fd96c0aecc0 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.659+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd95c02a070 con 0x7fd96c0aecc0 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.659+0000 7fd9617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.689 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.659+0000 7fd9617fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:24.742 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.741+0000 7fd973814700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fae1ca7c-8b05-406f-bfa7-c61093a32111", "id": 5}] 2023-09-30T01:03:24.742 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.741+0000 7fd973814700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fae1ca7c-8b05-406f-bfa7-c61093a32111", "id": 5}] 2023-09-30T01:03:24.743 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.741+0000 7fd973814700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.743 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.741+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fae1ca7c-8b05-406f-bfa7-c61093a32111", "id": 5} v 0) v1 -- 0x7fd96c072960 con 0x7fd96c0aecc0 2023-09-30T01:03:24.745 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.744+0000 7fd9617fa700 1 -- 172.21.15.157:0/3604000442 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fae1ca7c-8b05-406f-bfa7-c61093a32111", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd95c015020 con 0x7fd96c0aecc0 2023-09-30T01:03:24.745 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.744+0000 7fd9617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fae1ca7c-8b05-406f-bfa7-c61093a32111", "id": 5}] 2023-09-30T01:03:24.745 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.744+0000 7fd9617fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:24.745 INFO:teuthology.orchestra.run.smithi157.stdout:5 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 10 librados: watch_flush enter 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 10 librados: watch_flush exit 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 20 client.4150.objecter shutdown clearing up homeless session... 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 10 client.4150.objecter successfully canceled tick 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 10 monclient: shutdown 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 msgr2=0x7fd96c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009fd0 tx=0x7fd95c017000 comp rx=0 tx=0).stop 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 shutdown_connections 2023-09-30T01:03:24.747 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd96c1415c0 0x7fd96c0a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd96c0ae8f0 0x7fd96c0a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 --2- 172.21.15.157:0/3604000442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd96c0aecc0 0x7fd96c13dda0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 >> 172.21.15.157:0/3604000442 conn(0x7fd96c06a460 msgr2=0x7fd96c144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 shutdown_connections 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 -- 172.21.15.157:0/3604000442 wait complete. 2023-09-30T01:03:24.748 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.745+0000 7fd973814700 1 librados: shutdown 2023-09-30T01:03:24.758 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 5a22a721-1492-4746-9b2f-c28fd52c465f 6 2023-09-30T01:03:24.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.875+0000 7f5bef563700 10 monclient: get_monmap_and_config 2023-09-30T01:03:24.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.875+0000 7f5bef563700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.875+0000 7f5bef563700 10 monclient: monmap: 2023-09-30T01:03:24.877 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.877 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:24.877398+0000 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:24.877398+0000 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:24.878 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:24.879 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:24.879 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.879 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.876+0000 7f5bef563700 1 Processor -- start 2023-09-30T01:03:24.879 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.876+0000 7f5bef563700 1 -- start start 2023-09-30T01:03:24.879 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.876+0000 7f5bef563700 10 monclient: init 2023-09-30T01:03:24.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-09-30T01:03:24.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5be80ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:24.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5be80aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:24.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5be80a5d10 0x7f5be80adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-c con 0x7f5be80a5d10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be80a6190 con 0x7f5be80aecc0 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be80a7040 con 0x7f5be80ae8f0 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be8141cd0 con 0x7f5be80a5d10 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient(hunting): authenticate will time out at 1050.341119s 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bed2ff700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5be80ae8f0 0x7f5be80b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44590/0 (socket says 172.21.15.157:44590) 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bed2ff700 1 -- 172.21.15.157:0/3275839414 learned_addr learned my addr 172.21.15.157:0/3275839414 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 --2- 172.21.15.157:0/3275839414 >> v2:172.21.15.157:3300/0 conn(0x7f5be80a5d10 0x7f5be80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): get_auth_request con 0x7f5be80a5d10 auth_method 0 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 -- 172.21.15.157:0/3275839414 >> v1:172.21.15.157:6789/0 conn(0x7f5be80ae8f0 legacy=0x7f5be80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 -- 172.21.15.157:0/3275839414 >> v1:172.21.15.115:6789/0 conn(0x7f5be80aecc0 legacy=0x7f5be81415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.890 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 monclient: found mon.noname-c 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 -- 172.21.15.157:0/3275839414 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be80a57f0 con 0x7f5be80a5d10 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bedb00700 1 --2- 172.21.15.157:0/3275839414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be80adb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009e60 tx=0x7f5be0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e89052c9141889ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 1 -- 172.21.15.157:0/3275839414 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be000e070 con 0x7f5be80a5d10 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: dump: 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.891 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 1 -- 172.21.15.157:0/3275839414 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be000bc30 con 0x7f5be80a5d10 2023-09-30T01:03:24.892 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 5 monclient: authenticate success, global_id 4153 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 1 -- 172.21.15.157:0/3275839414 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0012430 con 0x7f5be80a5d10 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bef563700 4 set_mon_vals no callback set 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.877+0000 7f5bdffff700 10 monclient: dump: 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.893 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 10 monclient: shutdown 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 -- 172.21.15.157:0/3275839414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 msgr2=0x7f5be80adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 --2- 172.21.15.157:0/3275839414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be80adb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009e60 tx=0x7f5be0009780 comp rx=0 tx=0).stop 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 -- 172.21.15.157:0/3275839414 shutdown_connections 2023-09-30T01:03:24.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 --2- 172.21.15.157:0/3275839414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be80adb60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 -- 172.21.15.157:0/3275839414 >> 172.21.15.157:0/3275839414 conn(0x7f5be806a460 msgr2=0x7f5be80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 -- 172.21.15.157:0/3275839414 shutdown_connections 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.878+0000 7f5bef563700 1 -- 172.21.15.157:0/3275839414 wait complete. 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 10 monclient: build_initial_monmap 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 10 monclient: monmap: 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.895 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 librados: starting msgr at 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 librados: starting objecter 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 Processor -- start 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 -- start start 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 librados: setting wanted keys 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 1 librados: calling monclient init 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.879+0000 7f5bef563700 10 monclient: init 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:24.896 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be813e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-b con 0x7f5be80a5d10 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 0x7f5be80a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-a con 0x7f5be80ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5be80aecc0 0x7f5be8150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): picked mon.noname-c con 0x7f5be80aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80ab7d0 con 0x7f5be80ae8f0 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80a84b0 con 0x7f5be80a5d10 2023-09-30T01:03:24.897 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80a81c0 con 0x7f5be80aecc0 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bef563700 10 monclient(hunting): authenticate will time out at 1050.344314s 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 0x7f5be80a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be813e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57370/0 (socket says 172.21.15.157:57370) 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 1 -- 172.21.15.157:0/1305332645 learned_addr learned my addr 172.21.15.157:0/1305332645 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): get_auth_request con 0x7f5be80a5d10 auth_method 0 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.898 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): get_auth_request con 0x7f5be80ae8f0 auth_method 0 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5be80aecc0 0x7f5be8150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): get_auth_request con 0x7f5be80aecc0 auth_method 0 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.899 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bed2ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bedb00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 -- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5be80aecc0 msgr2=0x7f5be8150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5be80aecc0 0x7f5be8150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 -- 172.21.15.157:0/1305332645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 msgr2=0x7f5be813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.900 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 monclient: found mon.noname-a 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 -- 172.21.15.157:0/1305332645 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be0009a20 con 0x7f5be80ae8f0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5becafe700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 0x7f5be80a7c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c490 tx=0x7f5bd8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bdd7fa700 10 client.?.objecter ms_handle_connect 0x7f5be80ae8f0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.880+0000 7f5bdd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800ce40 con 0x7f5be80ae8f0 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:24.901 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: dump: 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:24.902 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 5 monclient: authenticate success, global_id 4155 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _renew_subs 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8011590 con 0x7f5be80ae8f0 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be813ee60 con 0x7f5be80ae8f0 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8011950 con 0x7f5be80ae8f0 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 monclient: dump: 2023-09-30T01:03:24.903 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:24.904 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _renew_subs 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdf7fe700 4 set_mon_vals no callback set 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be8150b20 con 0x7f5be80ae8f0 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 1 librados: init done 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5bd801c440 con 0x7f5be80ae8f0 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f5bd8000f90 con 0x7f5be80ae8f0 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 10 client.4155.objecter ms_dispatch 0x7f5be8001550 osd_map(8..8 src has 1..8) v4 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 3 client.4155.objecter handle_osd_map decoding full epoch 8 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bdd7fa700 20 client.4155.objecter dump_active .. 0 homeless 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.905 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.881+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be8072960 con 0x7f5be80ae8f0 2023-09-30T01:03:24.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.884+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5bd8017070 con 0x7f5be80ae8f0 2023-09-30T01:03:24.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.884+0000 7f5bdd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:24.906 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.884+0000 7f5bdd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:24.937 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:24.937+0000 7fa3f6d3d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:03:24.966 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.965+0000 7f5bef563700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5a22a721-1492-4746-9b2f-c28fd52c465f", "id": 6}] 2023-09-30T01:03:24.966 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.965+0000 7f5bef563700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5a22a721-1492-4746-9b2f-c28fd52c465f", "id": 6}] 2023-09-30T01:03:24.967 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.965+0000 7f5bef563700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:24.967 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.965+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5a22a721-1492-4746-9b2f-c28fd52c465f", "id": 6} v 0) v1 -- 0x7f5be80a54c0 con 0x7f5be80ae8f0 2023-09-30T01:03:24.969 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.968+0000 7f5bdd7fa700 1 -- 172.21.15.157:0/1305332645 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5a22a721-1492-4746-9b2f-c28fd52c465f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5be80a6190 con 0x7f5be80ae8f0 2023-09-30T01:03:24.969 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.968+0000 7f5bdd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5a22a721-1492-4746-9b2f-c28fd52c465f", "id": 6}] 2023-09-30T01:03:24.969 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.968+0000 7f5bdd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:24.969 INFO:teuthology.orchestra.run.smithi157.stdout:6 2023-09-30T01:03:24.970 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 10 librados: watch_flush enter 2023-09-30T01:03:24.970 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 10 librados: watch_flush exit 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 20 client.4155.objecter shutdown clearing up homeless session... 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 10 client.4155.objecter successfully canceled tick 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 10 monclient: shutdown 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 msgr2=0x7f5be80a7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 0x7f5be80a7c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c490 tx=0x7f5bd8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 shutdown_connections 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5be80aecc0 0x7f5be8150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5be80a5d10 0x7f5be813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.971 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 --2- 172.21.15.157:0/1305332645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5be80ae8f0 0x7f5be80a7c80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:24.972 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 >> 172.21.15.157:0/1305332645 conn(0x7f5be806a460 msgr2=0x7f5be80b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:24.972 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 shutdown_connections 2023-09-30T01:03:24.972 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 -- 172.21.15.157:0/1305332645 wait complete. 2023-09-30T01:03:24.972 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:24.969+0000 7f5bef563700 1 librados: shutdown 2023-09-30T01:03:24.981 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 2963067c-f278-4458-a7b2-9163558524b7 7 2023-09-30T01:03:25.009 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.009+0000 7fa3f6d3d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:03:25.060 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.059+0000 7fa6f95401c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:03:25.099 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.098+0000 7f93f61d2700 10 monclient: get_monmap_and_config 2023-09-30T01:03:25.099 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.098+0000 7f93f61d2700 10 monclient: build_initial_monmap 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.098+0000 7f93f61d2700 10 monclient: monmap: 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:03:25.100410+0000 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:03:25.100410+0000 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:25.100 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:25.101 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:25.101 INFO:teuthology.orchestra.run.smithi157.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:25.101 INFO:teuthology.orchestra.run.smithi157.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:25.101 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.111 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.099+0000 7f93f61d2700 1 Processor -- start 2023-09-30T01:03:25.111 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.099+0000 7f93f61d2700 1 -- start start 2023-09-30T01:03:25.111 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.099+0000 7f93f61d2700 10 monclient: init 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93f00ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f93f00aecc0 0x7f93f00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-a con 0x7f93f00aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93f01415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00a7040 con 0x7f93f00ae8f0 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00a57f0 con 0x7f93f01415c0 2023-09-30T01:03:25.112 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f00a5480 con 0x7f93f00aecc0 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93f61d2700 10 monclient(hunting): authenticate will time out at 1050.564313s 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93effff700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f93f01415c0 0x7f93f00adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44600/0 (socket says 172.21.15.157:44600) 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93effff700 1 -- 172.21.15.157:0/1456013879 learned_addr learned my addr 172.21.15.157:0/1456013879 (peer_addr_for_me v1:172.21.15.157:0/0) 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 1 --2- 172.21.15.157:0/1456013879 >> v2:172.21.15.115:3300/0 conn(0x7f93f00aecc0 0x7f93f00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): get_auth_request con 0x7f93f00aecc0 auth_method 0 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.113 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93eeffd700 1 -- 172.21.15.157:0/1456013879 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251572189 0 0) 0x7f93f00a57f0 con 0x7f93f01415c0 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93eeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93eeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93eeffd700 10 monclient(hunting): my global_id is 4162 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93eeffd700 1 -- 172.21.15.157:0/1456013879 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc003570 con 0x7f93f01415c0 2023-09-30T01:03:25.114 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient(hunting): handle_auth_done global_id 4158 payload 274 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 1 -- 172.21.15.157:0/1456013879 >> v1:172.21.15.157:6789/0 conn(0x7f93f01415c0 legacy=0x7f93f00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 1 -- 172.21.15.157:0/1456013879 >> v1:172.21.15.115:6789/0 conn(0x7f93f00ae8f0 legacy=0x7f93f00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 1 monclient: found mon.noname-a 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.100+0000 7f93e7fff700 1 -- 172.21.15.157:0/1456013879 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00a55c0 con 0x7f93f00aecc0 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93e7fff700 1 --2- 172.21.15.157:0/1456013879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f00aecc0 0x7f93f00a5d10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098e0 tx=0x7f93d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a540522faf2a48 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:25.115 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 1 -- 172.21.15.157:0/1456013879 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8011630 con 0x7f93f00aecc0 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: dump: 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.116 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 5 monclient: authenticate success, global_id 4158 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 1 -- 172.21.15.157:0/1456013879 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8011790 con 0x7f93f00aecc0 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:25.117 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 4 set_mon_vals no callback set 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 1 -- 172.21.15.157:0/1456013879 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8011b30 con 0x7f93f00aecc0 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93eeffd700 10 monclient: dump: 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.118 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 10 monclient: shutdown 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:25.119 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 -- 172.21.15.157:0/1456013879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f00aecc0 msgr2=0x7f93f00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 --2- 172.21.15.157:0/1456013879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f00aecc0 0x7f93f00a5d10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098e0 tx=0x7f93d80091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 -- 172.21.15.157:0/1456013879 shutdown_connections 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 --2- 172.21.15.157:0/1456013879 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f00aecc0 0x7f93f00a5d10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 -- 172.21.15.157:0/1456013879 >> 172.21.15.157:0/1456013879 conn(0x7f93f006a460 msgr2=0x7f93f00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 -- 172.21.15.157:0/1456013879 shutdown_connections 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.101+0000 7f93f61d2700 1 -- 172.21.15.157:0/1456013879 wait complete. 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 10 monclient: build_initial_monmap 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 10 monclient: monmap: 2023-09-30T01:03:25.120 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:25.121 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 librados: starting msgr at 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 librados: starting objecter 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 Processor -- start 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 -- start start 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 librados: setting wanted keys 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 1 librados: calling monclient init 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.102+0000 7f93f61d2700 10 monclient: init 2023-09-30T01:03:25.122 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-b con 0x7f93f00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f93f00aecc0 0x7f93f0151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-c con 0x7f93f00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 0x7f93f01504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): picked mon.noname-a con 0x7f93f01415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.123 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0151280 con 0x7f93f01415c0 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0151c50 con 0x7f93f00ae8f0 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0151dc0 con 0x7f93f00aecc0 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93f61d2700 10 monclient(hunting): authenticate will time out at 1050.567433s 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57382/0 (socket says 172.21.15.157:57382) 2023-09-30T01:03:25.124 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 -- 172.21.15.157:0/4022232304 learned_addr learned my addr 172.21.15.157:0/4022232304 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): get_auth_request con 0x7f93f00ae8f0 auth_method 0 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 0x7f93f01504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): get_auth_request con 0x7f93f01415c0 auth_method 0 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.125 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f93f00aecc0 0x7f93f0151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): get_auth_request con 0x7f93f00aecc0 auth_method 0 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.126 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 -- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f93f00aecc0 msgr2=0x7f93f0151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f93f00aecc0 0x7f93f0151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.127 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 -- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 msgr2=0x7f93f01504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 0x7f93f01504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 monclient: found mon.noname-b 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93ef7fe700 1 -- 172.21.15.157:0/4022232304 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8009420 con 0x7f93f00ae8f0 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.103+0000 7f93effff700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 0x7f93f01504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93ef7fe700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c810 tx=0x7f93e0007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 client.?.objecter ms_handle_connect 0x7f93f00ae8f0 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011590 con 0x7f93f00ae8f0 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: dump: 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 5 monclient: authenticate success, global_id 4165 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e0011730 con 0x7f93f00ae8f0 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011af0 con 0x7f93f00ae8f0 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 10 monclient: _renew_subs 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f0151f80 con 0x7f93f00ae8f0 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93ee7fc700 4 set_mon_vals no callback set 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 monclient: dump: 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr: 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 10 monclient: _renew_subs 2023-09-30T01:03:25.133 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f0150de0 con 0x7f93f00ae8f0 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93f61d2700 1 librados: init done 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f93e001c6c0 con 0x7f93f00ae8f0 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f93e0012a90 con 0x7f93f00ae8f0 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 10 client.4165.objecter ms_dispatch 0x7f93f0001550 osd_map(9..9 src has 1..9) v4 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 3 client.4165.objecter handle_osd_map got epochs [9,9] > 0 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 3 client.4165.objecter handle_osd_map decoding full epoch 9 2023-09-30T01:03:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.104+0000 7f93e77fe700 20 client.4165.objecter dump_active .. 0 homeless 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.105+0000 7f93f61d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.105+0000 7f93f61d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.105+0000 7f93f61d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.105+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f0072960 con 0x7f93f00ae8f0 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.107+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f93e0021060 con 0x7f93f00ae8f0 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.107+0000 7f93e77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.135 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.107+0000 7f93e77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:25.136 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.135+0000 7fa6f95401c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:03:25.141 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.141+0000 7fa3f6d3d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:03:25.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.189+0000 7f93f61d2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2963067c-f278-4458-a7b2-9163558524b7", "id": 7}] 2023-09-30T01:03:25.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.189+0000 7f93f61d2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2963067c-f278-4458-a7b2-9163558524b7", "id": 7}] 2023-09-30T01:03:25.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.189+0000 7f93f61d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:25.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.189+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2963067c-f278-4458-a7b2-9163558524b7", "id": 7} v 0) v1 -- 0x7f93f0141a40 con 0x7f93f00ae8f0 2023-09-30T01:03:25.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.192+0000 7f93e77fe700 1 -- 172.21.15.157:0/4022232304 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2963067c-f278-4458-a7b2-9163558524b7", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f93e002a0b0 con 0x7f93f00ae8f0 2023-09-30T01:03:25.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93e77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2963067c-f278-4458-a7b2-9163558524b7", "id": 7}] 2023-09-30T01:03:25.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93e77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:25.194 INFO:teuthology.orchestra.run.smithi157.stdout:7 2023-09-30T01:03:25.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 10 librados: watch_flush enter 2023-09-30T01:03:25.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 10 librados: watch_flush exit 2023-09-30T01:03:25.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 20 client.4165.objecter shutdown clearing up homeless session... 2023-09-30T01:03:25.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 10 client.4165.objecter successfully canceled tick 2023-09-30T01:03:25.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 10 monclient: shutdown 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 msgr2=0x7f93f013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c810 tx=0x7f93e0007f30 comp rx=0 tx=0).stop 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 shutdown_connections 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f93f01415c0 0x7f93f01504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93f00ae8f0 0x7f93f013e480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 --2- 172.21.15.157:0/4022232304 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f93f00aecc0 0x7f93f0151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.193+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 >> 172.21.15.157:0/4022232304 conn(0x7f93f006a460 msgr2=0x7f93f0144790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:25.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.194+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 shutdown_connections 2023-09-30T01:03:25.197 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.194+0000 7f93f61d2700 1 -- 172.21.15.157:0/4022232304 wait complete. 2023-09-30T01:03:25.197 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T01:03:25.194+0000 7f93f61d2700 1 librados: shutdown 2023-09-30T01:03:25.206 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T01:03:25.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T01:03:25.208 INFO:tasks.ceph.osd.0:Started 2023-09-30T01:03:25.209 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T01:03:25.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T01:03:25.209 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.208+0000 7fa3f6d3d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:03:25.211 INFO:tasks.ceph.osd.1:Started 2023-09-30T01:03:25.211 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T01:03:25.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T01:03:25.213 INFO:tasks.ceph.osd.2:Started 2023-09-30T01:03:25.213 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T01:03:25.213 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T01:03:25.215 INFO:tasks.ceph.osd.3:Started 2023-09-30T01:03:25.215 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T01:03:25.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T01:03:25.217 INFO:tasks.ceph.osd.4:Started 2023-09-30T01:03:25.218 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T01:03:25.218 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T01:03:25.219 INFO:tasks.ceph.osd.5:Started 2023-09-30T01:03:25.220 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T01:03:25.220 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T01:03:25.222 INFO:tasks.ceph.osd.6:Started 2023-09-30T01:03:25.222 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T01:03:25.222 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T01:03:25.224 INFO:tasks.ceph.osd.7:Started 2023-09-30T01:03:25.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:03:25.308 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.308+0000 7fa6f95401c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:03:25.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.381+0000 7f18b2421700 10 monclient: get_monmap_and_config 2023-09-30T01:03:25.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.381+0000 7f18b2421700 10 monclient: build_initial_monmap 2023-09-30T01:03:25.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.381+0000 7f18b2421700 10 monclient: monmap: 2023-09-30T01:03:25.381 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:25.382272+0000 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:25.382272+0000 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:25.382 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:25.383 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.383 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.381+0000 7fa6f95401c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 Processor -- start 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 -- start start 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient: init 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T01:03:25.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f18ac0ae8f0 0x7f18ac0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f18ac0aecc0 0x7f18ac0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18ac1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0a7040 con 0x7f18ac1415c0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ac0b19b0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ac0a57f0 con 0x7f18ac0ae8f0 2023-09-30T01:03:25.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18b2421700 10 monclient(hunting): authenticate will time out at 1066.832681s 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f18ac0aecc0 0x7f18ac0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f18ac0aecc0 0x7f18ac0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57672/0 (socket says 172.21.15.115:57672) 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 1 -- 172.21.15.115:0/4100202734 learned_addr learned my addr 172.21.15.115:0/4100202734 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): get_auth_request con 0x7f18ac0aecc0 auth_method 0 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18abfff700 1 --2- 172.21.15.115:0/4100202734 >> v2:172.21.15.157:3300/0 conn(0x7f18ac0ae8f0 0x7f18ac0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18abfff700 10 monclient(hunting): get_auth_request con 0x7f18ac0ae8f0 auth_method 0 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.382+0000 7f18abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_done global_id 4182 payload 274 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 -- 172.21.15.115:0/4100202734 >> v2:172.21.15.157:3300/0 conn(0x7f18ac0ae8f0 msgr2=0x7f18ac0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 --2- 172.21.15.115:0/4100202734 >> v2:172.21.15.157:3300/0 conn(0x7f18ac0ae8f0 0x7f18ac0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 -- 172.21.15.115:0/4100202734 >> v1:172.21.15.157:6789/0 conn(0x7f18ac1415c0 legacy=0x7f18ac0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 -- 172.21.15.115:0/4100202734 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0a5480 con 0x7f18ac0aecc0 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18ab7fe700 1 --2- 172.21.15.115:0/4100202734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac0b1470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b59c0 tx=0x7f18a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555e90aa915e6ef3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 1 -- 172.21.15.115:0/4100202734 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0009e70 con 0x7f18ac0aecc0 2023-09-30T01:03:25.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: dump: 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.396 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 1 -- 172.21.15.115:0/4100202734 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000b860 con 0x7f18ac0aecc0 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18b2421700 5 monclient: authenticate success, global_id 4182 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18b2421700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 1 -- 172.21.15.115:0/4100202734 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000bc00 con 0x7f18ac0aecc0 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18b2421700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18b2421700 4 set_mon_vals no callback set 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:25.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18aaffd700 10 monclient: dump: 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.398 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.383+0000 7f18b2421700 10 monclient: shutdown 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 -- 172.21.15.115:0/4100202734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 msgr2=0x7f18ac0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 --2- 172.21.15.115:0/4100202734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac0b1470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b59c0 tx=0x7f18a0009250 comp rx=0 tx=0).stop 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 -- 172.21.15.115:0/4100202734 shutdown_connections 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 --2- 172.21.15.115:0/4100202734 >> v2:172.21.15.157:3300/0 conn(0x7f18ac0ae8f0 0x7f18ac0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 --2- 172.21.15.115:0/4100202734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac0b1470 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 -- 172.21.15.115:0/4100202734 >> 172.21.15.115:0/4100202734 conn(0x7f18ac06a460 msgr2=0x7f18ac0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 -- 172.21.15.115:0/4100202734 shutdown_connections 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 -- 172.21.15.115:0/4100202734 wait complete. 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 10 monclient: build_initial_monmap 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 10 monclient: monmap: 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 librados: starting msgr at 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.384+0000 7f18b2421700 1 librados: starting objecter 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 Processor -- start 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 -- start start 2023-09-30T01:03:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 librados: setting wanted keys 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 librados: calling monclient init 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient: init 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 0x7f18ac13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac1415c0 0x7f18ac1518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): picked mon.noname-b con 0x7f18ac1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0abc20 con 0x7f18ac0aecc0 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0aa340 con 0x7f18ac1415c0 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0a88c0 con 0x7f18ac0ae8f0 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18b2421700 10 monclient(hunting): authenticate will time out at 1066.835830s 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57688/0 (socket says 172.21.15.115:57688) 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 1 -- 172.21.15.115:0/1716571651 learned_addr learned my addr 172.21.15.115:0/1716571651 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 10 monclient(hunting): get_auth_request con 0x7f18ac0aecc0 auth_method 0 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18abfff700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 0x7f18ac13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18abfff700 10 monclient(hunting): get_auth_request con 0x7f18ac0ae8f0 auth_method 0 2023-09-30T01:03:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.385+0000 7f18abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac1415c0 0x7f18ac1518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): get_auth_request con 0x7f18ac1415c0 auth_method 0 2023-09-30T01:03:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b09be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient(hunting): handle_auth_done global_id 4185 payload 824 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 -- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 msgr2=0x7f18ac13e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 0x7f18ac13e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 -- 172.21.15.115:0/1716571651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac1415c0 msgr2=0x7f18ac1518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac1415c0 0x7f18ac1518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 -- 172.21.15.115:0/1716571651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00094f0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18abfff700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 0x7f18ac13e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18ab7fe700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b0920 tx=0x7f18a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 client.?.objecter ms_handle_connect 0x7f18ac0aecc0 2023-09-30T01:03:25.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000bbe0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: dump: 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.409 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: _finish_auth 0 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 5 monclient: authenticate success, global_id 4185 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 10 monclient: _renew_subs 2023-09-30T01:03:25.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0011420 con 0x7f18ac0aecc0 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 1 -- 172.21.15.115:0/1716571651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac0a94e0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a00117c0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 monclient: dump: 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18aa7fc700 4 set_mon_vals no callback set 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 10 monclient: _renew_subs 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 1 -- 172.21.15.115:0/1716571651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac0aa9e0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f18b2421700 1 librados: init done 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f18a0011bf0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f18a0023ad0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 10 client.4185.objecter ms_dispatch 0x7f18ac001550 osd_map(10..10 src has 1..10) v4 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 3 client.4185.objecter handle_osd_map decoding full epoch 10 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.386+0000 7f1893fff700 20 client.4185.objecter dump_active .. 0 homeless 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.387+0000 7f1891ffb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.387+0000 7f1891ffb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.387+0000 7f1891ffb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.387+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18840050b0 con 0x7f18ac0aecc0 2023-09-30T01:03:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.389+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f18a0028070 con 0x7f18ac0aecc0 2023-09-30T01:03:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.389+0000 7f1893fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.389+0000 7f1893fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:25.415 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.416+0000 7fa3f6d3d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:03:25.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.457+0000 7f1891ffb700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:25.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.457+0000 7f1891ffb700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:25.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.457+0000 7f1891ffb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:25.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.457+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1884005630 con 0x7f18ac0aecc0 2023-09-30T01:03:25.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.458+0000 7f1893fff700 1 -- 172.21.15.115:0/1716571651 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f18a0015020 con 0x7f18ac0aecc0 2023-09-30T01:03:25.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.458+0000 7f1893fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:25.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.458+0000 7f1893fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:25.460 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:25.460 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":10,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:25.192334+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:03:25.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 10 librados: watch_flush enter 2023-09-30T01:03:25.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 10 librados: watch_flush exit 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 20 client.4185.objecter shutdown clearing up homeless session... 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 10 client.4185.objecter successfully canceled tick 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 10 monclient: shutdown 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 msgr2=0x7f18ac151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0b0920 tx=0x7f18a0017000 comp rx=0 tx=0).stop 2023-09-30T01:03:25.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 shutdown_connections 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac1415c0 0x7f18ac1518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f18ac0ae8f0 0x7f18ac13e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 --2- 172.21.15.115:0/1716571651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18ac0aecc0 0x7f18ac151360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 >> 172.21.15.115:0/1716571651 conn(0x7f18ac06a460 msgr2=0x7f18ac144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 shutdown_connections 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 -- 172.21.15.115:0/1716571651 wait complete. 2023-09-30T01:03:25.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:25.459+0000 7f1891ffb700 1 librados: shutdown 2023-09-30T01:03:25.471 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T01:03:25.471 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T01:03:25.494 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.494+0000 7fa3f6d3d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:03:25.563 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.562+0000 7fa6f95401c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:03:25.571 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.571+0000 7fa3f6d3d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:03:25.642 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.641+0000 7fa6f95401c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:03:25.667 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.666+0000 7fa3f6d3d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:03:25.718 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.717+0000 7fa6f95401c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:03:25.806 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.807+0000 7fa6f95401c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:03:25.814 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T01:03:25.814+0000 7feebbf66540 -1 Falling back to public interface 2023-09-30T01:03:25.821 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T01:03:25.821+0000 7ffba0cfb540 -1 Falling back to public interface 2023-09-30T01:03:25.831 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T01:03:25.831+0000 7f56b4da3540 -1 Falling back to public interface 2023-09-30T01:03:25.834 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T01:03:25.834+0000 7f5313906540 -1 Falling back to public interface 2023-09-30T01:03:25.835 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T01:03:25.835+0000 7fd2b4e7d540 -1 Falling back to public interface 2023-09-30T01:03:25.836 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T01:03:25.836+0000 7fae52f1e540 -1 Falling back to public interface 2023-09-30T01:03:25.840 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T01:03:25.840+0000 7fdb2837b540 -1 Falling back to public interface 2023-09-30T01:03:25.841 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T01:03:25.842+0000 7f56b017a540 -1 Falling back to public interface 2023-09-30T01:03:25.843 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.843+0000 7fa3f6d3d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:03:25.923 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:25.923+0000 7fa3f6d3d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:03:25.990 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:25.990+0000 7fa6f95401c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:03:26.069 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:26.070+0000 7fa6f95401c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:03:26.205 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:26.205+0000 7fa3f6d3d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:03:26.273 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:26.273+0000 7fa3f6d3d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:03:26.277 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:03:26.359 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:26.359+0000 7fa6f95401c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:03:26.387 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.387+0000 7f48ab365700 10 monclient: get_monmap_and_config 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.387+0000 7f48ab365700 10 monclient: build_initial_monmap 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.387+0000 7f48ab365700 10 monclient: monmap: 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:epoch 0 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:03:26.388642+0000 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:03:26.388642+0000 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:26.388 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:26.389 INFO:teuthology.misc.health.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:26.389 INFO:teuthology.misc.health.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:26.389 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.388+0000 7f48ab365700 1 Processor -- start 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.388+0000 7f48ab365700 1 -- start start 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.388+0000 7f48ab365700 10 monclient: init 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.388+0000 7f48ab365700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-b con 0x7f48a40a56d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48a4141540 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f48a413ad60 0x7f48a413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:26.398 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-c con 0x7f48a413ad60 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a40a7d00 con 0x7f48a4141540 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a40a8190 con 0x7f48a413ad60 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a41420c0 con 0x7f48a40a56d0 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient(hunting): authenticate will time out at 1067.839123s 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34816/0 (socket says 172.21.15.115:34816) 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 -- 172.21.15.115:0/3646069509 learned_addr learned my addr 172.21.15.115:0/3646069509 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): get_auth_request con 0x7f48a40a56d0 auth_method 0 2023-09-30T01:03:26.399 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 1 --2- 172.21.15.115:0/3646069509 >> v2:172.21.15.157:3300/0 conn(0x7f48a413ad60 0x7f48a413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): get_auth_request con 0x7f48a413ad60 auth_method 0 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:26.400 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 -- 172.21.15.115:0/3646069509 >> v2:172.21.15.157:3300/0 conn(0x7f48a413ad60 msgr2=0x7f48a413b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 --2- 172.21.15.115:0/3646069509 >> v2:172.21.15.157:3300/0 conn(0x7f48a413ad60 0x7f48a413b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 1 -- 172.21.15.115:0/3646069509 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200363902 0 0) 0x7f48a40a7d00 con 0x7f48a4141540 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 -- 172.21.15.115:0/3646069509 >> v1:172.21.15.157:6789/0 conn(0x7f48a4141540 legacy=0x7f48a41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient: _finish_hunting 0 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 monclient: found mon.noname-b 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 -- 172.21.15.115:0/3646069509 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40a6f60 con 0x7f48a40a56d0 2023-09-30T01:03:26.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48a9101700 1 --2- 172.21.15.115:0/3646069509 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f48a00099b0 tx=0x7f48a0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32738efab7927c9b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 1 -- 172.21.15.115:0/3646069509 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0009ea0 con 0x7f48a40a56d0 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: dump: 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.402 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 1 -- 172.21.15.115:0/3646069509 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a000b860 con 0x7f48a40a56d0 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 1 -- 172.21.15.115:0/3646069509 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a000bbe0 con 0x7f48a40a56d0 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 5 monclient: authenticate success, global_id 4160 2023-09-30T01:03:26.403 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f48ab365700 4 set_mon_vals no callback set 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.389+0000 7f489bfff700 10 monclient: dump: 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:26.404 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 10 monclient: shutdown 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 -- 172.21.15.115:0/3646069509 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 msgr2=0x7f48a40a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 --2- 172.21.15.115:0/3646069509 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f48a00099b0 tx=0x7f48a0009250 comp rx=0 tx=0).stop 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 -- 172.21.15.115:0/3646069509 shutdown_connections 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 --2- 172.21.15.115:0/3646069509 >> v2:172.21.15.157:3300/0 conn(0x7f48a413ad60 0x7f48a413b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 --2- 172.21.15.115:0/3646069509 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a40a56d0 0x7f48a40a5aa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.405 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 -- 172.21.15.115:0/3646069509 >> 172.21.15.115:0/3646069509 conn(0x7f48a4063280 msgr2=0x7f48a4063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 -- 172.21.15.115:0/3646069509 shutdown_connections 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.390+0000 7f48ab365700 1 -- 172.21.15.115:0/3646069509 wait complete. 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 10 monclient: build_initial_monmap 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 10 monclient: monmap: 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:epoch 0 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:created 0.000000 2023-09-30T01:03:26.406 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 librados: starting msgr at 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 librados: starting objecter 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 Processor -- start 2023-09-30T01:03:26.407 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 -- start start 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 librados: setting wanted keys 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 1 librados: calling monclient init 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.391+0000 7f48ab365700 10 monclient: init 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-a con 0x7f48a40a56d0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 0x7f48a40ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-c con 0x7f48a413ad60 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 0x7f48a4151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:26.408 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): picked mon.noname-b con 0x7f48a4141540 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a413c580 con 0x7f48a40a56d0 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a40ae710 con 0x7f48a4141540 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a4151450 con 0x7f48a413ad60 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48ab365700 10 monclient(hunting): authenticate will time out at 1067.842373s 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 0x7f48a40ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57696/0 (socket says 172.21.15.115:57696) 2023-09-30T01:03:26.409 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 -- 172.21.15.115:0/426057459 learned_addr learned my addr 172.21.15.115:0/426057459 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): get_auth_request con 0x7f48a40a56d0 auth_method 0 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): get_auth_request con 0x7f48a413ad60 auth_method 0 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 0x7f48a4151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): get_auth_request con 0x7f48a4141540 auth_method 0 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:26.410 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2023-09-30T01:03:26.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 -- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 msgr2=0x7f48a40ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 0x7f48a40ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 -- 172.21.15.115:0/426057459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 msgr2=0x7f48a4151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 0x7f48a4151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient: _finish_hunting 0 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 monclient: found mon.noname-a 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 -- 172.21.15.115:0/426057459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00094f0 con 0x7f48a40a56d0 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a8900700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 0x7f48a40ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9902700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 0x7f48a4151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:26.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48a9101700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48a4142200 tx=0x7f48a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48997fa700 10 client.?.objecter ms_handle_connect 0x7f48a40a56d0 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.392+0000 7f48997fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a000bbe0 con 0x7f48a40a56d0 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: dump: 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.413 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 5 monclient: authenticate success, global_id 4188 2023-09-30T01:03:26.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a0000d10 con 0x7f48a40a56d0 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 10 monclient: _renew_subs 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a4151730 con 0x7f48a40a56d0 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0011650 con 0x7f48a40a56d0 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 monclient: dump: 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.415 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f489b7fe700 4 set_mon_vals no callback set 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 10 monclient: _renew_subs 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a40adc40 con 0x7f48a40a56d0 2023-09-30T01:03:26.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48ab365700 1 librados: init done 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f48a0011d00 con 0x7f48a40a56d0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f48a00253f0 con 0x7f48a40a56d0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 10 client.4188.objecter ms_dispatch 0x7f48a4001550 osd_map(10..10 src has 1..10) v4 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 3 client.4188.objecter handle_osd_map decoding full epoch 10 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.393+0000 7f48997fa700 20 client.4188.objecter dump_active .. 0 homeless 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.394+0000 7f48ab365700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.394+0000 7f48ab365700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.394+0000 7f48ab365700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.394+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a40a7410 con 0x7f48a40a56d0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.396+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f48a0033060 con 0x7f48a40a56d0 2023-09-30T01:03:26.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.396+0000 7f48997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:26.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.396+0000 7f48997fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:26.430 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:26.431+0000 7fa6f95401c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:03:26.463 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.462+0000 7f48ab365700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:26.463 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.462+0000 7f48ab365700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:26.463 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.462+0000 7f48ab365700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:26.463 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.462+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48a41387b0 con 0x7f48a40a56d0 2023-09-30T01:03:26.463 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.463+0000 7f48997fa700 1 -- 172.21.15.115:0/426057459 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f48a0015020 con 0x7f48a40a56d0 2023-09-30T01:03:26.464 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.463+0000 7f48997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:26.464 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.463+0000 7f48997fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:26.464 INFO:teuthology.misc.health.smithi115.stdout: 2023-09-30T01:03:26.464 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":10,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:25.192334+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:03:26.465 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 10 librados: watch_flush enter 2023-09-30T01:03:26.465 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 10 librados: watch_flush exit 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 20 client.4188.objecter shutdown clearing up homeless session... 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 10 client.4188.objecter successfully canceled tick 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 10 monclient: shutdown 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 msgr2=0x7f48a413fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48a4142200 tx=0x7f48a0017000 comp rx=0 tx=0).stop 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 shutdown_connections 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48a4141540 0x7f48a4151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48a40a56d0 0x7f48a413fa90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 --2- 172.21.15.115:0/426057459 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f48a413ad60 0x7f48a40ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:26.466 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 >> 172.21.15.115:0/426057459 conn(0x7f48a4063280 msgr2=0x7f48a40b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:26.467 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 shutdown_connections 2023-09-30T01:03:26.467 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 -- 172.21.15.115:0/426057459 wait complete. 2023-09-30T01:03:26.467 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:26.464+0000 7f48ab365700 1 librados: shutdown 2023-09-30T01:03:26.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T01:03:26.788 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:26.788+0000 7fa3f6d3d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:03:26.938 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:26.938+0000 7fa6f95401c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:03:27.096 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.096+0000 7fa3f6d3d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:03:27.236 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.236+0000 7fa3f6d3d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:03:27.257 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:27.258+0000 7fa6f95401c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:03:27.398 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:27.398+0000 7fa6f95401c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:03:27.591 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.591+0000 7fa3f6d3d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:03:27.733 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.733+0000 7fa3f6d3d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:03:27.758 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:27.759+0000 7fa6f95401c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:03:27.881 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.881+0000 7fa3f6d3d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:03:27.902 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:27.902+0000 7fa6f95401c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:03:27.981 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:27.981+0000 7fa3f6d3d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:03:28.024 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:28.024+0000 7fa3861e7700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:03:28.024 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:28.024+0000 7fa3861e7700 -1 pg_autoscaler.serve: 2023-09-30T01:03:28.025 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:28.024+0000 7fa3861e7700 -1 Traceback (most recent call last): 2023-09-30T01:03:28.025 INFO:tasks.ceph.mgr.y.smithi115.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T01:03:28.025 INFO:tasks.ceph.mgr.y.smithi115.stderr: self._maybe_adjust() 2023-09-30T01:03:28.025 INFO:tasks.ceph.mgr.y.smithi115.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T01:03:28.025 INFO:tasks.ceph.mgr.y.smithi115.stderr: if self.noautoscale: 2023-09-30T01:03:28.026 INFO:tasks.ceph.mgr.y.smithi115.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:03:28.026 INFO:tasks.ceph.mgr.y.smithi115.stderr: 2023-09-30T01:03:28.054 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:28.054+0000 7fa6f95401c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:03:28.154 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T01:03:28.154+0000 7fa6f95401c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:03:29.007 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:29.007+0000 7fa3bb421700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T01:03:29.007 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T01:03:29.007+0000 7fa3bb421700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T01:03:29.306 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T01:03:29.305+0000 7feebbf66540 -1 osd.0 0 log_to_monitors true 2023-09-30T01:03:29.314 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T01:03:29.314+0000 7ffba0cfb540 -1 osd.4 0 log_to_monitors true 2023-09-30T01:03:29.341 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T01:03:29.341+0000 7f56b4da3540 -1 osd.2 0 log_to_monitors true 2023-09-30T01:03:29.344 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T01:03:29.345+0000 7f5313906540 -1 osd.5 0 log_to_monitors true 2023-09-30T01:03:29.345 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T01:03:29.345+0000 7fd2b4e7d540 -1 osd.6 0 log_to_monitors true 2023-09-30T01:03:29.364 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T01:03:29.365+0000 7f56b017a540 -1 osd.7 0 log_to_monitors true 2023-09-30T01:03:29.366 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T01:03:29.366+0000 7fae52f1e540 -1 osd.1 0 log_to_monitors true 2023-09-30T01:03:29.375 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T01:03:29.375+0000 7fdb2837b540 -1 osd.3 0 log_to_monitors true 2023-09-30T01:03:31.010 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T01:03:31.010+0000 7ffb97900700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T01:03:31.011 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T01:03:31.011+0000 7f56a6d7f700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T01:03:31.011 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T01:03:31.012+0000 7fd2ad285700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T01:03:31.012 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T01:03:31.012+0000 7f530a50b700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T01:03:31.012 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T01:03:31.011+0000 7fdb1ef80700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T01:03:31.016 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T01:03:31.017+0000 7ffb92efc700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.017 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T01:03:31.017+0000 7f56a237b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.019 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T01:03:31.019+0000 7f56ab9a8700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T01:03:31.020 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T01:03:31.019+0000 7feeb436e700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T01:03:31.020 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T01:03:31.020+0000 7fae49b23700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T01:03:31.020 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T01:03:31.020+0000 7f5305b07700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.021 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T01:03:31.020+0000 7fd2a707e700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.023 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T01:03:31.023+0000 7fdb1a57c700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.027 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T01:03:31.027+0000 7fae4511f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.027 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T01:03:31.027+0000 7f56a6fa4700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:31.027 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T01:03:31.027+0000 7feeae167700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:03:33.018 INFO:tasks.ceph.mon.a.smithi115.stderr:2023-09-30T01:03:33.017+0000 7f2a84da5700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:03:33.281 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:03:33.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.401+0000 7fa947da6700 10 monclient: get_monmap_and_config 2023-09-30T01:03:33.401 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.401+0000 7fa947da6700 10 monclient: build_initial_monmap 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.401+0000 7fa947da6700 10 monclient: monmap: 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:epoch 0 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:03:33.402595+0000 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:03:33.402595+0000 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:33.402 INFO:teuthology.misc.health.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:33.403 INFO:teuthology.misc.health.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:33.403 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.402+0000 7fa947da6700 1 Processor -- start 2023-09-30T01:03:33.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.402+0000 7fa947da6700 1 -- start start 2023-09-30T01:03:33.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.402+0000 7fa947da6700 10 monclient: init 2023-09-30T01:03:33.411 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.402+0000 7fa947da6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 0x7fa9400a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-c con 0x7fa9400a56d0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa940141540 0x7fa9401419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-a con 0x7fa940141540 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9400af220 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9400a7d00 con 0x7fa9400af220 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9400a8190 con 0x7fa940141540 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9400a6f60 con 0x7fa9400a56d0 2023-09-30T01:03:33.412 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient(hunting): authenticate will time out at 1074.853115s 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa940141540 0x7fa9401419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa940141540 0x7fa9401419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43906/0 (socket says 172.21.15.115:43906) 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 -- 172.21.15.115:0/837257826 learned_addr learned my addr 172.21.15.115:0/837257826 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): get_auth_request con 0x7fa940141540 auth_method 0 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.413 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945b42700 1 --2- 172.21.15.115:0/837257826 >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 0x7fa9400a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 -- 172.21.15.115:0/837257826 >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 msgr2=0x7fa9400a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 --2- 172.21.15.115:0/837257826 >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 0x7fa9400a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 -- 172.21.15.115:0/837257826 >> v1:172.21.15.157:6789/0 conn(0x7fa9400af220 legacy=0x7fa9400af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient: _finish_hunting 0 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 monclient: found mon.noname-a 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 -- 172.21.15.115:0/837257826 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9400a7950 con 0x7fa940141540 2023-09-30T01:03:33.414 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945b42700 10 monclient: get_auth_request con 0x7fa9400a56d0 auth_method 0 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945b42700 1 --2- 172.21.15.115:0/837257826 >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 0x7fa9400a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa945341700 1 --2- 172.21.15.115:0/837257826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa940141540 0x7fa9401419b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9400649e0 tx=0x7fa930009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea752ebc3ccd78f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 1 -- 172.21.15.115:0/837257826 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000b820 con 0x7fa940141540 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: dump: 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.415 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: _finish_auth 0 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 1 -- 172.21.15.115:0/837257826 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93000b980 con 0x7fa940141540 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 5 monclient: authenticate success, global_id 4230 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 1 -- 172.21.15.115:0/837257826 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000bd20 con 0x7fa940141540 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.416 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa944b40700 10 monclient: dump: 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:33.417 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.403+0000 7fa947da6700 4 set_mon_vals no callback set 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 10 monclient: shutdown 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 -- 172.21.15.115:0/837257826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa940141540 msgr2=0x7fa9401419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 --2- 172.21.15.115:0/837257826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa940141540 0x7fa9401419b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9400649e0 tx=0x7fa930009250 comp rx=0 tx=0).stop 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 -- 172.21.15.115:0/837257826 shutdown_connections 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 --2- 172.21.15.115:0/837257826 >> v2:172.21.15.157:3300/0 conn(0x7fa9400a56d0 0x7fa9400a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 --2- 172.21.15.115:0/837257826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa940141540 0x7fa9401419b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 -- 172.21.15.115:0/837257826 >> 172.21.15.115:0/837257826 conn(0x7fa940063280 msgr2=0x7fa940063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 -- 172.21.15.115:0/837257826 shutdown_connections 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.404+0000 7fa947da6700 1 -- 172.21.15.115:0/837257826 wait complete. 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient: build_initial_monmap 2023-09-30T01:03:33.418 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient: monmap: 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:epoch 0 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:created 0.000000 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 librados: starting msgr at 2023-09-30T01:03:33.419 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 librados: starting objecter 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 Processor -- start 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 -- start start 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 librados: setting wanted keys 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 librados: calling monclient init 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient: init 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 0x7fa940140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.405+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-c con 0x7fa9400a56d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-a con 0x7fa9400af220 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:33.420 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa940141540 0x7fa94014d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): picked mon.noname-b con 0x7fa940141540 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa940140ef0 con 0x7fa9400af220 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94013b640 con 0x7fa940141540 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94013a0d0 con 0x7fa9400a56d0 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient(hunting): authenticate will time out at 1074.856046s 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 0x7fa940140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.421 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43916/0 (socket says 172.21.15.115:43916) 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 1 -- 172.21.15.115:0/1282083092 learned_addr learned my addr 172.21.15.115:0/1282083092 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): get_auth_request con 0x7fa9400af220 auth_method 0 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): get_auth_request con 0x7fa9400a56d0 auth_method 0 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.422 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa940141540 0x7fa94014d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): get_auth_request con 0x7fa940141540 auth_method 0 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa946343700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2023-09-30T01:03:33.423 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 -- 172.21.15.115:0/1282083092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa940141540 msgr2=0x7fa94014d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa940141540 0x7fa94014d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 -- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 msgr2=0x7fa94013aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient: _finish_hunting 0 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 monclient: found mon.noname-c 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 -- 172.21.15.115:0/1282083092 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9300094f0 con 0x7fa9400a56d0 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945341700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa945b42700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 0x7fa940140220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c880 tx=0x7fa93c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 client.?.objecter ms_handle_connect 0x7fa9400a56d0 2023-09-30T01:03:33.424 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c011600 con 0x7fa9400a56d0 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: dump: 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.425 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 5 monclient: authenticate success, global_id 4202 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c011760 con 0x7fa9400a56d0 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c011b20 con 0x7fa9400a56d0 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient: _renew_subs 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94013a290 con 0x7fa9400a56d0 2023-09-30T01:03:33.426 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa935ffb700 10 monclient: dump: 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:epoch 1 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr: 2023-09-30T01:03:33.427 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.406+0000 7fa937fff700 4 set_mon_vals no callback set 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.407+0000 7fa947da6700 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.407+0000 7fa947da6700 10 monclient: _renew_subs 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.407+0000 7fa947da6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.407+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94014de10 con 0x7fa9400a56d0 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.407+0000 7fa947da6700 1 librados: init done 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa947da6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa947da6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa947da6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94013b490 con 0x7fa9400a56d0 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fa93c018360 con 0x7fa9400a56d0 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 1 --2- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 0x7fa92406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.428 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3722+0+0 (secure 0 0 0) 0x7fa93c083a70 con 0x7fa9400a56d0 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 10 client.4202.objecter ms_dispatch 0x7fa940001550 osd_map(14..14 src has 1..14) v4 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 3 client.4202.objecter handle_osd_map got epochs [14,14] > 0 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 3 client.4202.objecter handle_osd_map decoding full epoch 14 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa945341700 1 --2- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 0x7fa92406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa935ffb700 20 client.4202.objecter dump_active .. 0 homeless 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.408+0000 7fa945341700 10 monclient: get_auth_request con 0x7fa924068800 auth_method 0 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.409+0000 7fa935ffb700 10 client.4202.objecter ms_handle_connect 0x7fa924068800 2023-09-30T01:03:33.429 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.409+0000 7fa945341700 1 --2- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 0x7fa92406acb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9400abdb0 tx=0x7fa930016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.430 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.413+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa93c051580 con 0x7fa9400a56d0 2023-09-30T01:03:33.430 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.413+0000 7fa935ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.430 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.413+0000 7fa935ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:33.600 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa947da6700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:33.600 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa947da6700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:33.600 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa947da6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:33.600 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9400ab570 con 0x7fa9400a56d0 2023-09-30T01:03:33.601 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa935ffb700 1 -- 172.21.15.115:0/1282083092 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10451 (secure 0 0 0) 0x7fa93c01f090 con 0x7fa9400a56d0 2023-09-30T01:03:33.601 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa935ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:33.601 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.600+0000 7fa935ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:33.601 INFO:teuthology.misc.health.smithi115.stdout: 2023-09-30T01:03:33.601 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":14,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:33.014324+0000","last_up_change":"2023-09-30T01:03:32.012648+0000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:03:32.045422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6801","nonce":115935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6803","nonce":115935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115935}]},"public_addr":"172.21.15.115:6800/115935","cluster_addr":"172.21.15.115:6801/115935","heartbeat_back_addr":"172.21.15.115:6803/115935","heartbeat_front_addr":"172.21.15.115:6802/115935","state":["exists","up"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":115950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6811","nonce":115950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115950}]},"public_addr":"172.21.15.115:6808/115950","cluster_addr":"172.21.15.115:6809/115950","heartbeat_back_addr":"172.21.15.115:6811/115950","heartbeat_front_addr":"172.21.15.115:6810/115950","state":["exists","up"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6805","nonce":115951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6807","nonce":115951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115951}]},"public_addr":"172.21.15.115:6804/115951","cluster_addr":"172.21.15.115:6805/115951","heartbeat_back_addr":"172.21.15.115:6807/115951","heartbeat_front_addr":"172.21.15.115:6806/115951","state":["exists","up"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6813","nonce":115952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":115952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115952}]},"public_addr":"172.21.15.115:6812/115952","cluster_addr":"172.21.15.115:6813/115952","heartbeat_back_addr":"172.21.15.115:6815/115952","heartbeat_front_addr":"172.21.15.115:6814/115952","state":["exists","up"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6801","nonce":115846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6803","nonce":115846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115846}]},"public_addr":"172.21.15.157:6800/115846","cluster_addr":"172.21.15.157:6801/115846","heartbeat_back_addr":"172.21.15.157:6803/115846","heartbeat_front_addr":"172.21.15.157:6802/115846","state":["exists","up"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6805","nonce":115847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":115847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115847}]},"public_addr":"172.21.15.157:6804/115847","cluster_addr":"172.21.15.157:6805/115847","heartbeat_back_addr":"172.21.15.157:6807/115847","heartbeat_front_addr":"172.21.15.157:6806/115847","state":["exists","up"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":115848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6811","nonce":115848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115848}]},"public_addr":"172.21.15.157:6808/115848","cluster_addr":"172.21.15.157:6809/115848","heartbeat_back_addr":"172.21.15.157:6811/115848","heartbeat_front_addr":"172.21.15.157:6810/115848","state":["exists","up"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6815","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115891}]},"public_addr":"172.21.15.157:6812/115891","cluster_addr":"172.21.15.157:6813/115891","heartbeat_back_addr":"172.21.15.157:6815/115891","heartbeat_front_addr":"172.21.15.157:6814/115891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.324933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.319648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.364624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.391390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.296016+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.336020+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.397884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.386449+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 10 librados: watch_flush enter 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 10 librados: watch_flush exit 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 20 client.4202.objecter shutdown clearing up homeless session... 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 10 client.4202.objecter successfully canceled tick 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 msgr2=0x7fa92406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 0x7fa92406acb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9400abdb0 tx=0x7fa930016000 comp rx=0 tx=0).stop 2023-09-30T01:03:33.603 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 10 monclient: shutdown 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 msgr2=0x7fa940140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 0x7fa940140220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c880 tx=0x7fa93c008040 comp rx=0 tx=0).stop 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 shutdown_connections 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> v2:172.21.15.115:6816/115635 conn(0x7fa924068800 0x7fa92406acb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa940141540 0x7fa94014d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa9400a56d0 0x7fa940140220 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 --2- 172.21.15.115:0/1282083092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa9400af220 0x7fa94013aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 >> 172.21.15.115:0/1282083092 conn(0x7fa940063280 msgr2=0x7fa940063650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 shutdown_connections 2023-09-30T01:03:33.604 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.602+0000 7fa947da6700 1 -- 172.21.15.115:0/1282083092 wait complete. 2023-09-30T01:03:33.605 INFO:teuthology.misc.health.smithi115.stderr:2023-09-30T01:03:33.603+0000 7fa947da6700 1 librados: shutdown 2023-09-30T01:03:33.614 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T01:03:33.614 INFO:tasks.ceph:Creating RBD pool 2023-09-30T01:03:33.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T01:03:33.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 10 monclient: get_monmap_and_config 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 10 monclient: build_initial_monmap 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 10 monclient: monmap: 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:33.731164+0000 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:33.731164+0000 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:33.742 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 1 Processor -- start 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 1 -- start start 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.730+0000 7ff85045f700 10 monclient: init 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8480ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8480aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:33.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff8480a5d10 0x7ff8480adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-c con 0x7ff8480a5d10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480a6190 con 0x7ff8480ae8f0 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480a7040 con 0x7ff8480aecc0 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff848141cd0 con 0x7ff8480a5d10 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff85045f700 10 monclient(hunting): authenticate will time out at 1075.181528s 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e1fb700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff8480ae8f0 0x7ff8480b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46558/0 (socket says 172.21.15.115:46558) 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e1fb700 1 -- 172.21.15.115:0/192960582 learned_addr learned my addr 172.21.15.115:0/192960582 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218606252 0 0) 0x7ff844001110 con 0x7ff8480ae8f0 2023-09-30T01:03:33.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84d1f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84d1f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84d1f9700 10 monclient(hunting): my global_id is 4236 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff834003570 con 0x7ff8480ae8f0 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/192960582 >> v2:172.21.15.157:3300/0 conn(0x7ff8480a5d10 0x7ff8480adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request con 0x7ff8480a5d10 auth_method 0 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e9fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.731+0000 7ff84e9fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711491516 0 0) 0x7ff834003570 con 0x7ff8480ae8f0 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 >> v2:172.21.15.157:3300/0 conn(0x7ff8480a5d10 msgr2=0x7ff8480adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 --2- 172.21.15.115:0/192960582 >> v2:172.21.15.157:3300/0 conn(0x7ff8480a5d10 0x7ff8480adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 >> v1:172.21.15.157:6789/0 conn(0x7ff8480aecc0 legacy=0x7ff8481415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: _finish_hunting 0 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480a57f0 con 0x7ff8480ae8f0 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: _finish_auth 0 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:33.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 5 monclient: authenticate success, global_id 4236 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7ff844001600 con 0x7ff8480ae8f0 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: dump: 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.747 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:33.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 >> v1:172.21.15.115:6789/0 conn(0x7ff8480ae8f0 legacy=0x7ff8480b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff834003a60 0x7ff834003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): picked mon.b con 0x7ff834003a60 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): picked mon.c con 0x7ff834004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): picked mon.a con 0x7ff8340051c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff848141cd0 con 0x7ff8340051c0 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83400c0d0 con 0x7ff834003a60 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83400c290 con 0x7ff834004810 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request con 0x7ff8340051c0 auth_method 0 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff834003a60 0x7ff834003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): get_auth_request con 0x7ff834004810 auth_method 0 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): get_auth_request con 0x7ff834003a60 auth_method 0 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_done global_id 4236 payload 293 2023-09-30T01:03:33.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 -- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff834004810 msgr2=0x7ff834004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 -- 172.21.15.115:0/192960582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff834003a60 msgr2=0x7ff834003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff834003a60 0x7ff834003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 monclient: found mon.a 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 -- 172.21.15.115:0/192960582 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834003570 con 0x7ff8340051c0 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 -- 172.21.15.115:0/192960582 reap_dead start 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e210 tx=0x7ff84000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96e09a888c36350 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000eac0 con 0x7ff8340051c0 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: dump: 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.754 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84000f040 con 0x7ff8340051c0 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 1 -- 172.21.15.115:0/192960582 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000ee70 con 0x7ff8340051c0 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff85045f700 4 set_mon_vals no callback set 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.732+0000 7ff84d1f9700 10 monclient: dump: 2023-09-30T01:03:33.755 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 10 monclient: shutdown 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 -- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 msgr2=0x7ff83400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e210 tx=0x7ff84000b040 comp rx=0 tx=0).stop 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 -- 172.21.15.115:0/192960582 shutdown_connections 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 --2- 172.21.15.115:0/192960582 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 -- 172.21.15.115:0/192960582 >> 172.21.15.115:0/192960582 conn(0x7ff84806a460 msgr2=0x7ff8480630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 -- 172.21.15.115:0/192960582 shutdown_connections 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.733+0000 7ff85045f700 1 -- 172.21.15.115:0/192960582 wait complete. 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 10 monclient: build_initial_monmap 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 10 monclient: monmap: 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:33.757 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 librados: starting msgr at 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 librados: starting objecter 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 Processor -- start 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 -- start start 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 librados: setting wanted keys 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 1 librados: calling monclient init 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.734+0000 7ff85045f700 10 monclient: init 2023-09-30T01:03:33.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340051c0 0x7ff84813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-b con 0x7ff8340051c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-a con 0x7ff848151b70 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 0x7ff848138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): picked mon.noname-c con 0x7ff848138ac0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480aa980 con 0x7ff848151b70 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84813d990 con 0x7ff8340051c0 2023-09-30T01:03:33.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480aabf0 con 0x7ff848138ac0 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff85045f700 10 monclient(hunting): authenticate will time out at 1075.185703s 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 0x7ff848138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43932/0 (socket says 172.21.15.115:43932) 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 1 -- 172.21.15.115:0/476398826 learned_addr learned my addr 172.21.15.115:0/476398826 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): get_auth_request con 0x7ff848151b70 auth_method 0 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request con 0x7ff848138ac0 auth_method 0 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340051c0 0x7ff84813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e9fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): get_auth_request con 0x7ff8340051c0 auth_method 0 2023-09-30T01:03:33.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.735+0000 7ff84e1fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 -- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 msgr2=0x7ff848138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 0x7ff848138f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 -- 172.21.15.115:0/476398826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340051c0 msgr2=0x7ff84813d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340051c0 0x7ff84813d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:33.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 10 monclient: _finish_hunting 0 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 monclient: found mon.noname-a 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 -- 172.21.15.115:0/476398826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480a7970 con 0x7ff848151b70 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84e9fc700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 0x7ff848138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84d9fa700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff838002950 tx=0x7ff838007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 client.?.objecter ms_handle_connect 0x7ff848151b70 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838003390 con 0x7ff848151b70 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: dump: 2023-09-30T01:03:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff838013600 con 0x7ff848151b70 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 5 monclient: authenticate success, global_id 4239 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 10 monclient: _renew_subs 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8480a8590 con 0x7ff848151b70 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380139a0 con 0x7ff848151b70 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff83e7fc700 10 monclient: dump: 2023-09-30T01:03:33.765 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff84c9f8700 4 set_mon_vals no callback set 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:33.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 10 monclient: _renew_subs 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff848139450 con 0x7ff848151b70 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.736+0000 7ff85045f700 1 librados: init done 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.737+0000 7ff85045f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.737+0000 7ff85045f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.737+0000 7ff85045f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.737+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff848141390 con 0x7ff848151b70 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.737+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7ff838013d50 con 0x7ff848151b70 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 1 --2- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 0x7ff82c06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3722+0+0 (secure 0 0 0) 0x7ff838083820 con 0x7ff848151b70 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 10 client.4239.objecter ms_dispatch 0x7ff848001550 osd_map(14..14 src has 1..14) v4 2023-09-30T01:03:33.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 3 client.4239.objecter handle_osd_map got epochs [14,14] > 0 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff84e1fb700 1 --2- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 0x7ff82c06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 3 client.4239.objecter handle_osd_map decoding full epoch 14 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff84e1fb700 10 monclient: get_auth_request con 0x7ff82c068800 auth_method 0 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 20 client.4239.objecter dump_active .. 0 homeless 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff83e7fc700 10 client.4239.objecter ms_handle_connect 0x7ff82c068800 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.738+0000 7ff84e1fb700 1 --2- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 0x7ff82c06acb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff844007b60 tx=0x7ff844007a50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:33.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.742+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff838051330 con 0x7ff848151b70 2023-09-30T01:03:33.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.742+0000 7ff83e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:33.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.742+0000 7ff83e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:33.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.923+0000 7ff85045f700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:03:33.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.923+0000 7ff85045f700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:03:33.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.923+0000 7ff85045f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:33.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:33.923+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff848072960 con 0x7ff848151b70 2023-09-30T01:03:34.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.023+0000 7ff83e7fc700 1 -- 172.21.15.115:0/476398826 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7ff838054950 con 0x7ff848151b70 2023-09-30T01:03:34.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.023+0000 7ff83e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:03:34.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.023+0000 7ff83e7fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T01:03:34.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 10 librados: watch_flush enter 2023-09-30T01:03:34.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 10 librados: watch_flush exit 2023-09-30T01:03:34.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 20 client.4239.objecter shutdown clearing up homeless session... 2023-09-30T01:03:34.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 10 client.4239.objecter successfully canceled tick 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 msgr2=0x7ff82c06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 0x7ff82c06acb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff844007b60 tx=0x7ff844007a50 comp rx=0 tx=0).stop 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 10 monclient: shutdown 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 msgr2=0x7ff8480a9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff838002950 tx=0x7ff838007d30 comp rx=0 tx=0).stop 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 shutdown_connections 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> v2:172.21.15.115:6816/115635 conn(0x7ff82c068800 0x7ff82c06acb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff848138ac0 0x7ff848138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340051c0 0x7ff84813d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 --2- 172.21.15.115:0/476398826 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff848151b70 0x7ff8480a9340 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 >> 172.21.15.115:0/476398826 conn(0x7ff84806a460 msgr2=0x7ff84813ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:34.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 shutdown_connections 2023-09-30T01:03:34.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 -- 172.21.15.115:0/476398826 wait complete. 2023-09-30T01:03:34.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.027+0000 7ff85045f700 1 librados: shutdown 2023-09-30T01:03:34.029 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'rbd' created 2023-09-30T01:03:34.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.170+0000 7f9a8267f700 10 monclient: get_monmap_and_config 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.170+0000 7f9a8267f700 10 monclient: build_initial_monmap 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.170+0000 7f9a8267f700 10 monclient: monmap: 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:34.171898+0000 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:34.171898+0000 2023-09-30T01:03:34.171 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:34.172 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.171+0000 7f9a8267f700 1 Processor -- start 2023-09-30T01:03:34.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.171+0000 7f9a8267f700 1 -- start start 2023-09-30T01:03:34.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.171+0000 7f9a8267f700 10 monclient: init 2023-09-30T01:03:34.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a7c0a5640 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9a7c13e040 0x7f9a7c13e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c13e040 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a7c13acd0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0a8220 con 0x7f9a7c13acd0 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0a6dc0 con 0x7f9a7c0a5640 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c0a7d20 con 0x7f9a7c13e040 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a8267f700 10 monclient(hunting): authenticate will time out at 1075.622340s 2023-09-30T01:03:34.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a80c1c700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9a7c13acd0 0x7f9a7c13b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46560/0 (socket says 172.21.15.115:46560) 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a80c1c700 1 -- 172.21.15.115:0/3252857455 learned_addr learned my addr 172.21.15.115:0/3252857455 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 1 --2- 172.21.15.115:0/3252857455 >> v2:172.21.15.157:3300/0 conn(0x7f9a7c13e040 0x7f9a7c13e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): get_auth_request con 0x7f9a7c13e040 auth_method 0 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7affd700 1 -- 172.21.15.115:0/3252857455 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173911906 0 0) 0x7f9a7c0a8220 con 0x7f9a7c13acd0 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7affd700 10 monclient(hunting): my global_id is 4248 2023-09-30T01:03:34.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7affd700 1 -- 172.21.15.115:0/3252857455 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003570 con 0x7f9a7c13acd0 2023-09-30T01:03:34.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:34.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:34.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:34.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/3252857455 >> v1:172.21.15.157:6789/0 conn(0x7f9a7c0a5640 legacy=0x7f9a7c0a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/3252857455 >> v1:172.21.15.115:6789/0 conn(0x7f9a7c13acd0 legacy=0x7f9a7c13b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 1 monclient: found mon.noname-c 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:34.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.172+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/3252857455 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0a5a10 con 0x7f9a7c13e040 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7b7fe700 1 --2- 172.21.15.115:0/3252857455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c13e040 0x7f9a7c13e4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a700098e0 tx=0x7f9a700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb9fea4cf15e21e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 1 -- 172.21.15.115:0/3252857455 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7000b640 con 0x7f9a7c13e040 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: dump: 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:34.185 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: _finish_auth 0 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 5 monclient: authenticate success, global_id 4264 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 1 -- 172.21.15.115:0/3252857455 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7000b7a0 con 0x7f9a7c13e040 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:34.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 4 set_mon_vals no callback set 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 1 -- 172.21.15.115:0/3252857455 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7000bb40 con 0x7f9a7c13e040 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a7affd700 10 monclient: dump: 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:34.187 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 10 monclient: shutdown 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 -- 172.21.15.115:0/3252857455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c13e040 msgr2=0x7f9a7c13e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 --2- 172.21.15.115:0/3252857455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c13e040 0x7f9a7c13e4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a700098e0 tx=0x7f9a700091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 -- 172.21.15.115:0/3252857455 shutdown_connections 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 --2- 172.21.15.115:0/3252857455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c13e040 0x7f9a7c13e4c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 -- 172.21.15.115:0/3252857455 >> 172.21.15.115:0/3252857455 conn(0x7f9a7c068e40 msgr2=0x7f9a7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 -- 172.21.15.115:0/3252857455 shutdown_connections 2023-09-30T01:03:34.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.173+0000 7f9a8267f700 1 -- 172.21.15.115:0/3252857455 wait complete. 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 10 monclient: build_initial_monmap 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 10 monclient: monmap: 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:34.189 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 librados: starting msgr at 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 librados: starting objecter 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 Processor -- start 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 -- start start 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 librados: setting wanted keys 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 1 librados: calling monclient init 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.174+0000 7f9a8267f700 10 monclient: init 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 0x7f9a7c0adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:34.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c0a5640 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c13acd0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9a7c13e040 0x7f9a7c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c13e040 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0ae7f0 con 0x7f9a7c13acd0 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0ae360 con 0x7f9a7c0a5640 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c135b50 con 0x7f9a7c13e040 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a8267f700 10 monclient(hunting): authenticate will time out at 1075.625482s 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43956/0 (socket says 172.21.15.115:43956) 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/1731212991 learned_addr learned my addr 172.21.15.115:0/1731212991 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): get_auth_request con 0x7f9a7c13acd0 auth_method 0 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9a7c13e040 0x7f9a7c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): get_auth_request con 0x7f9a7c13e040 auth_method 0 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:34.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a80c1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 0x7f9a7c0adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 10 monclient(hunting): get_auth_request con 0x7f9a7c0a5640 auth_method 0 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:34.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9a7c13e040 msgr2=0x7f9a7c0ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9a7c13e040 0x7f9a7c0ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/1731212991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 msgr2=0x7f9a7c0adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 0x7f9a7c0adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7b7fe700 1 -- 172.21.15.115:0/1731212991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70009420 con 0x7f9a7c13acd0 2023-09-30T01:03:34.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.175+0000 7f9a7bfff700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 0x7f9a7c0adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a7b7fe700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c069540 tx=0x7f9a70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 client.?.objecter ms_handle_connect 0x7f9a7c13acd0 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70000d10 con 0x7f9a7c13acd0 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: dump: 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.195 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: _finish_auth 0 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 5 monclient: authenticate success, global_id 4251 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7000bb20 con 0x7f9a7c13acd0 2023-09-30T01:03:34.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70011650 con 0x7f9a7c13acd0 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a63fff700 10 monclient: dump: 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:34.197 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a7a7fc700 4 set_mon_vals no callback set 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 10 monclient: _renew_subs 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c135dd0 con 0x7f9a7c13acd0 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 10 monclient: _renew_subs 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c136290 con 0x7f9a7c13acd0 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.176+0000 7f9a8267f700 1 librados: init done 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.177+0000 7f9a8267f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:34.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.177+0000 7f9a8267f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.177+0000 7f9a8267f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.177+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c13fff0 con 0x7f9a7c13acd0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.177+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f9a70011a00 con 0x7f9a7c13acd0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 1 --2- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 0x7f9a6806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4075+0+0 (secure 0 0 0) 0x7f9a70085360 con 0x7f9a7c13acd0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a7bfff700 1 --2- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 0x7f9a6806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 10 client.4251.objecter ms_dispatch 0x7f9a7c001550 osd_map(15..15 src has 1..15) v4 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 3 client.4251.objecter handle_osd_map got epochs [15,15] > 0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 3 client.4251.objecter handle_osd_map decoding full epoch 15 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a7bfff700 10 monclient: get_auth_request con 0x7f9a68068800 auth_method 0 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 20 client.4251.objecter dump_active .. 0 homeless 2023-09-30T01:03:34.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a63fff700 10 client.4251.objecter ms_handle_connect 0x7f9a68068800 2023-09-30T01:03:34.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.178+0000 7f9a7bfff700 1 --2- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 0x7f9a6806acb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c007fd0 tx=0x7f9a6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:34.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.183+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a70052d10 con 0x7f9a7c13acd0 2023-09-30T01:03:34.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.183+0000 7f9a63fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:34.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.183+0000 7f9a63fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:34.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.384+0000 7f9a8267f700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:03:34.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.384+0000 7f9a8267f700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:03:34.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.385+0000 7f9a8267f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:34.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:34.385+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9a7c0729d0 con 0x7f9a7c13acd0 2023-09-30T01:03:35.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.029+0000 7f9a63fff700 1 -- 172.21.15.115:0/1731212991 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9a7c0729d0 con 0x7f9a7c13acd0 2023-09-30T01:03:35.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.029+0000 7f9a63fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:03:35.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.029+0000 7f9a63fff700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T01:03:35.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 10 librados: watch_flush enter 2023-09-30T01:03:35.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 10 librados: watch_flush exit 2023-09-30T01:03:35.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 20 client.4251.objecter shutdown clearing up homeless session... 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 10 client.4251.objecter successfully canceled tick 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 msgr2=0x7f9a6806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 0x7f9a6806acb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c007fd0 tx=0x7f9a6c009040 comp rx=0 tx=0).stop 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 10 monclient: shutdown 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 msgr2=0x7f9a7c0b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c069540 tx=0x7f9a70017000 comp rx=0 tx=0).stop 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 shutdown_connections 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> v2:172.21.15.115:6816/115635 conn(0x7f9a68068800 0x7f9a6806acb0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c007fd0 tx=0x7f9a6c009040 comp rx=0 tx=0).stop 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9a7c13e040 0x7f9a7c0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a7c0a5640 0x7f9a7c0adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 --2- 172.21.15.115:0/1731212991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9a7c13acd0 0x7f9a7c0b48d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 >> 172.21.15.115:0/1731212991 conn(0x7f9a7c068e40 msgr2=0x7f9a7c139260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 shutdown_connections 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 -- 172.21.15.115:0/1731212991 wait complete. 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.032+0000 7f9a8267f700 1 librados: shutdown 2023-09-30T01:03:35.034 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T01:03:35.045 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T01:03:35.045 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T01:03:35.045 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T01:03:35.045 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T01:03:35.046 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.162+0000 7fd1e8389700 10 monclient: get_monmap_and_config 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.162+0000 7fd1e8389700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.162+0000 7fd1e8389700 10 monclient: monmap: 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:35.163884+0000 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:35.163884+0000 2023-09-30T01:03:35.163 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:35.164 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.163+0000 7fd1e8389700 1 Processor -- start 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.163+0000 7fd1e8389700 1 -- start start 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.163+0000 7fd1e8389700 10 monclient: init 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:35.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1e00aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00a7040 con 0x7fd1e00aecc0 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e00a57f0 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e00a5480 con 0x7fd1e01415c0 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e8389700 10 monclient(hunting): authenticate will time out at 1076.614326s 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e5924700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd1e00aecc0 0x7fd1e00b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46570/0 (socket says 172.21.15.115:46570) 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58860/0 (socket says 172.21.15.115:58860) 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43984/0 (socket says 172.21.15.115:43984) 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e5924700 1 -- 172.21.15.115:0/2117645091 learned_addr learned my addr 172.21.15.115:0/2117645091 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): get_auth_request con 0x7fd1e00ae8f0 auth_method 0 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): get_auth_request con 0x7fd1e01415c0 auth_method 0 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 -- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 msgr2=0x7fd1e00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 -- 172.21.15.115:0/2117645091 >> v1:172.21.15.115:6789/0 conn(0x7fd1e00aecc0 legacy=0x7fd1e00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 monclient: found mon.noname-a 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e5123700 1 -- 172.21.15.115:0/2117645091 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665463806 0 0) 0x7fd1e00a7040 con 0x7fd1e00aecc0 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 -- 172.21.15.115:0/2117645091 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0141a40 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e5123700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6926700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.164+0000 7fd1e6125700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40098e0 tx=0x7fd1d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e5fd30830baec5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 1 -- 172.21.15.115:0/2117645091 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d400b640 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:35.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: dump: 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 5 monclient: authenticate success, global_id 4257 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 1 -- 172.21.15.115:0/2117645091 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d400b7a0 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 4 set_mon_vals no callback set 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 1 -- 172.21.15.115:0/2117645091 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d400bb40 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e5123700 10 monclient: dump: 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.180 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 10 monclient: shutdown 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 -- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00ae8f0 msgr2=0x7fd1e00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40098e0 tx=0x7fd1d40091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 -- 172.21.15.115:0/2117645091 shutdown_connections 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 --2- 172.21.15.115:0/2117645091 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e00a5d10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 -- 172.21.15.115:0/2117645091 >> 172.21.15.115:0/2117645091 conn(0x7fd1e006a460 msgr2=0x7fd1e00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 -- 172.21.15.115:0/2117645091 shutdown_connections 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.165+0000 7fd1e8389700 1 -- 172.21.15.115:0/2117645091 wait complete. 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 10 monclient: monmap: 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:35.182 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 librados: starting msgr at 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 librados: starting objecter 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 Processor -- start 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 -- start start 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 librados: setting wanted keys 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 1 librados: calling monclient init 2023-09-30T01:03:35.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.166+0000 7fd1e8389700 10 monclient: init 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e013d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): picked mon.noname-c con 0x7fd1e01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e00a7c30 con 0x7fd1e00aecc0 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0151670 con 0x7fd1e00ae8f0 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0151300 con 0x7fd1e01415c0 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient(hunting): authenticate will time out at 1076.617291s 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58870/0 (socket says 172.21.15.115:58870) 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:43988/0 (socket says 172.21.15.115:43988) 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 1 -- 172.21.15.115:0/1506053796 learned_addr learned my addr 172.21.15.115:0/1506053796 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): get_auth_request con 0x7fd1e01415c0 auth_method 0 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): get_auth_request con 0x7fd1e00aecc0 auth_method 0 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6125700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e013d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 -- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 msgr2=0x7fd1e00a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 -- 172.21.15.115:0/1506053796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 msgr2=0x7fd1e013d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e013d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 monclient: found mon.noname-a 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 -- 172.21.15.115:0/1506053796 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8009a20 con 0x7fd1e00aecc0 2023-09-30T01:03:35.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6125700 10 monclient: get_auth_request con 0x7fd1e00ae8f0 auth_method 0 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6125700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e013d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e6926700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e5924700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0011e0 tx=0x7fd1dc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 client.?.objecter ms_handle_connect 0x7fd1e00aecc0 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc005b30 con 0x7fd1e00aecc0 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: dump: 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.188 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc005c90 con 0x7fd1e00aecc0 2023-09-30T01:03:35.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc007300 con 0x7fd1e00aecc0 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1d27fc700 10 monclient: dump: 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.190 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 5 monclient: authenticate success, global_id 4263 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e4922700 4 set_mon_vals no callback set 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient: _renew_subs 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.167+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d4009420 con 0x7fd1e00aecc0 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:35.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 monclient: _renew_subs 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e00a86b0 con 0x7fd1e00aecc0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 1 librados: init done 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.168+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e0072960 con 0x7fd1e00aecc0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fd1dc01b3b0 con 0x7fd1e00aecc0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 1 --2- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 0x7fd1c406ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1e6125700 1 --2- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 0x7fd1c406ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1e6125700 10 monclient: get_auth_request con 0x7fd1c4068880 auth_method 0 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd1dc083370 con 0x7fd1e00aecc0 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 10 client.4263.objecter ms_dispatch 0x7fd1e0001550 osd_map(16..16 src has 1..16) v4 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 3 client.4263.objecter handle_osd_map got epochs [16,16] > 0 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.169+0000 7fd1d27fc700 3 client.4263.objecter handle_osd_map decoding full epoch 16 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.170+0000 7fd1d27fc700 20 client.4263.objecter dump_active .. 0 homeless 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.170+0000 7fd1d27fc700 10 client.4263.objecter ms_handle_connect 0x7fd1c4068880 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.170+0000 7fd1e6125700 1 --2- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 0x7fd1c406ad30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4003080 tx=0x7fd1d4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.174+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1dc050c70 con 0x7fd1e00aecc0 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.174+0000 7fd1d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.174+0000 7fd1d27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:35.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.405+0000 7fd1e8389700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:03:35.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.405+0000 7fd1e8389700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:03:35.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.405+0000 7fd1e8389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.405+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd1e01517b0 con 0x7fd1e00aecc0 2023-09-30T01:03:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.408+0000 7fd1d27fc700 1 -- 172.21.15.115:0/1506053796 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+163558 (secure 0 0 0) 0x7fd1dc017020 con 0x7fd1e00aecc0 2023-09-30T01:03:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.408+0000 7fd1d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:03:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.408+0000 7fd1d27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 10 librados: watch_flush enter 2023-09-30T01:03:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 10 librados: watch_flush exit 2023-09-30T01:03:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 20 client.4263.objecter shutdown clearing up homeless session... 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 10 client.4263.objecter successfully canceled tick 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 msgr2=0x7fd1c406ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 0x7fd1c406ad30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4003080 tx=0x7fd1d4017000 comp rx=0 tx=0).stop 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 10 monclient: shutdown 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 msgr2=0x7fd1e0151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0011e0 tx=0x7fd1dc007d30 comp rx=0 tx=0).stop 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 shutdown_connections 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> v2:172.21.15.115:6816/115635 conn(0x7fd1c4068880 0x7fd1c406ad30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e01415c0 0x7fd1e00a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e00ae8f0 0x7fd1e013d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 --2- 172.21.15.115:0/1506053796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e00aecc0 0x7fd1e0151b00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 >> 172.21.15.115:0/1506053796 conn(0x7fd1e006a460 msgr2=0x7fd1e00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 shutdown_connections 2023-09-30T01:03:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 -- 172.21.15.115:0/1506053796 wait complete. 2023-09-30T01:03:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.411+0000 7fd1e8389700 1 librados: shutdown 2023-09-30T01:03:35.423 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":4,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115635}]},"active_addr":"172.21.15.115:6816/115635","active_change":"2023-09-30T01:03:27.995546+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2423754513}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":3205769132}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2769990664}]}]} 2023-09-30T01:03:35.428 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T01:03:35.428 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T01:03:35.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.544+0000 7f52f873c700 10 monclient: get_monmap_and_config 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.544+0000 7f52f873c700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.544+0000 7f52f873c700 10 monclient: monmap: 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:35.545815+0000 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:35.545815+0000 2023-09-30T01:03:35.545 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:35.546 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.545+0000 7f52f873c700 1 Processor -- start 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.545+0000 7f52f873c700 1 -- start start 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.545+0000 7f52f873c700 10 monclient: init 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f52f00ae8f0 0x7f52f00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-c con 0x7f52f00ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-b con 0x7f52f00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:35.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 0x7f52f00adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-a con 0x7f52f01415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00a7040 con 0x7f52f01415c0 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00b19b0 con 0x7f52f00ae8f0 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00a57f0 con 0x7f52f00aecc0 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 10 monclient(hunting): authenticate will time out at 1076.996294s 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58876/0 (socket says 172.21.15.115:58876) 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 -- 172.21.15.115:0/2704057192 learned_addr learned my addr 172.21.15.115:0/2704057192 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): get_auth_request con 0x7f52f00aecc0 auth_method 0 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 0x7f52f00adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): get_auth_request con 0x7f52f01415c0 auth_method 0 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f64d8700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.157:3300/0 conn(0x7f52f00ae8f0 0x7f52f00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2023-09-30T01:03:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 -- 172.21.15.115:0/2704057192 >> v2:172.21.15.157:3300/0 conn(0x7f52f00ae8f0 msgr2=0x7f52f00a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.157:3300/0 conn(0x7f52f00ae8f0 0x7f52f00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 -- 172.21.15.115:0/2704057192 >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 msgr2=0x7f52f00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 0x7f52f00adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 monclient: found mon.noname-b 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 -- 172.21.15.115:0/2704057192 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00a5480 con 0x7f52f00aecc0 2023-09-30T01:03:35.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f6cd9700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 0x7f52f00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/2704057192 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52f00b59c0 tx=0x7f52ec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f11e0a99921f6655 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 1 -- 172.21.15.115:0/2704057192 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec009e70 con 0x7f52f00aecc0 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 10 monclient: dump: 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.561 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 5 monclient: authenticate success, global_id 4226 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 1 -- 172.21.15.115:0/2704057192 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec00b860 con 0x7f52f00aecc0 2023-09-30T01:03:35.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f873c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.546+0000 7f52f54d6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f54d6700 1 -- 172.21.15.115:0/2704057192 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec00bc00 con 0x7f52f00aecc0 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 4 set_mon_vals no callback set 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f54d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f54d6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f54d6700 10 monclient: dump: 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.563 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 10 monclient: shutdown 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 -- 172.21.15.115:0/2704057192 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 msgr2=0x7f52f00b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 --2- 172.21.15.115:0/2704057192 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52f00b59c0 tx=0x7f52ec009250 comp rx=0 tx=0).stop 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 -- 172.21.15.115:0/2704057192 shutdown_connections 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.115:3300/0 conn(0x7f52f01415c0 0x7f52f00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 --2- 172.21.15.115:0/2704057192 >> v2:172.21.15.157:3300/0 conn(0x7f52f00ae8f0 0x7f52f00a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 --2- 172.21.15.115:0/2704057192 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f00aecc0 0x7f52f00b1470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 -- 172.21.15.115:0/2704057192 >> 172.21.15.115:0/2704057192 conn(0x7f52f006a460 msgr2=0x7f52f00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 -- 172.21.15.115:0/2704057192 shutdown_connections 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.547+0000 7f52f873c700 1 -- 172.21.15.115:0/2704057192 wait complete. 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 10 monclient: monmap: 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.565 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 librados: starting msgr at 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 librados: starting objecter 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 Processor -- start 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 -- start start 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 librados: setting wanted keys 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 1 librados: calling monclient init 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.548+0000 7f52f873c700 10 monclient: init 2023-09-30T01:03:35.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52f00ae8f0 0x7f52f013e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-b con 0x7f52f00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-a con 0x7f52f00aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 0x7f52f014e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): picked mon.noname-c con 0x7f52f01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00aa9e0 con 0x7f52f00aecc0 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0151ef0 con 0x7f52f00ae8f0 2023-09-30T01:03:35.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00a8700 con 0x7f52f01415c0 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient(hunting): authenticate will time out at 1076.999197s 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 0x7f52f014e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44004/0 (socket says 172.21.15.115:44004) 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 -- 172.21.15.115:0/561190945 learned_addr learned my addr 172.21.15.115:0/561190945 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): get_auth_request con 0x7f52f00aecc0 auth_method 0 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): get_auth_request con 0x7f52f01415c0 auth_method 0 2023-09-30T01:03:35.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52f00ae8f0 0x7f52f013e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): get_auth_request con 0x7f52f00ae8f0 auth_method 0 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f64d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 -- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 msgr2=0x7f52f014e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 0x7f52f014e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 -- 172.21.15.115:0/561190945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52f00ae8f0 msgr2=0x7f52f013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52f00ae8f0 0x7f52f013e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 monclient: found mon.noname-a 2023-09-30T01:03:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 -- 172.21.15.115:0/561190945 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009a20 con 0x7f52f00aecc0 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f6cd9700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 0x7f52f014e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f5cd7700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52f00a5930 tx=0x7f52ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 client.?.objecter ms_handle_connect 0x7f52f00aecc0 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec000d10 con 0x7f52f00aecc0 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: dump: 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.572 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 5 monclient: authenticate success, global_id 4269 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec00bbe0 con 0x7f52f00aecc0 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec011670 con 0x7f52f00aecc0 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _renew_subs 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec0094f0 con 0x7f52f00aecc0 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f4cd5700 4 set_mon_vals no callback set 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:35.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52e67fc700 10 monclient: dump: 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:35.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _renew_subs 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f00ad610 con 0x7f52f00aecc0 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.549+0000 7f52f873c700 1 librados: init done 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.550+0000 7f52f873c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.550+0000 7f52f873c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.550+0000 7f52f873c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.550+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f0152280 con 0x7f52f00aecc0 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f52ec011a60 con 0x7f52f00aecc0 2023-09-30T01:03:35.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 1 --2- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 0x7f52d406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f52ec0851d0 con 0x7f52f00aecc0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 10 client.4269.objecter ms_dispatch 0x7f52f0001550 osd_map(16..16 src has 1..16) v4 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 3 client.4269.objecter handle_osd_map got epochs [16,16] > 0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52f64d8700 1 --2- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 0x7f52d406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 3 client.4269.objecter handle_osd_map decoding full epoch 16 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52f64d8700 10 monclient: get_auth_request con 0x7f52d4068800 auth_method 0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 20 client.4269.objecter dump_active .. 0 homeless 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52e67fc700 10 client.4269.objecter ms_handle_connect 0x7f52d4068800 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.551+0000 7f52f64d8700 1 --2- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 0x7f52d406acb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52e000acb0 tx=0x7f52e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.555+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52ec052c30 con 0x7f52f00aecc0 2023-09-30T01:03:35.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.555+0000 7f52e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.555+0000 7f52e67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:35.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.738+0000 7f52f873c700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:35.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.738+0000 7f52f873c700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:35.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.738+0000 7f52f873c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.738+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f0152060 con 0x7f52f00aecc0 2023-09-30T01:03:35.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.739+0000 7f52e67fc700 1 -- 172.21.15.115:0/561190945 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11980 (secure 0 0 0) 0x7f52ec056250 con 0x7f52f00aecc0 2023-09-30T01:03:35.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.739+0000 7f52e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:35.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.739+0000 7f52e67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:35.740 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:35.740 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":16,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:35.022958+0000","last_up_change":"2023-09-30T01:03:32.012648+0000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:03:32.045422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:03:33.941118+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6801","nonce":115935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6803","nonce":115935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115935}]},"public_addr":"172.21.15.115:6800/115935","cluster_addr":"172.21.15.115:6801/115935","heartbeat_back_addr":"172.21.15.115:6803/115935","heartbeat_front_addr":"172.21.15.115:6802/115935","state":["exists","up"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":115950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6811","nonce":115950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115950}]},"public_addr":"172.21.15.115:6808/115950","cluster_addr":"172.21.15.115:6809/115950","heartbeat_back_addr":"172.21.15.115:6811/115950","heartbeat_front_addr":"172.21.15.115:6810/115950","state":["exists","up"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6805","nonce":115951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6807","nonce":115951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115951}]},"public_addr":"172.21.15.115:6804/115951","cluster_addr":"172.21.15.115:6805/115951","heartbeat_back_addr":"172.21.15.115:6807/115951","heartbeat_front_addr":"172.21.15.115:6806/115951","state":["exists","up"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6813","nonce":115952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":115952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115952}]},"public_addr":"172.21.15.115:6812/115952","cluster_addr":"172.21.15.115:6813/115952","heartbeat_back_addr":"172.21.15.115:6815/115952","heartbeat_front_addr":"172.21.15.115:6814/115952","state":["exists","up"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6801","nonce":115846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6803","nonce":115846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115846}]},"public_addr":"172.21.15.157:6800/115846","cluster_addr":"172.21.15.157:6801/115846","heartbeat_back_addr":"172.21.15.157:6803/115846","heartbeat_front_addr":"172.21.15.157:6802/115846","state":["exists","up"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6805","nonce":115847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":115847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115847}]},"public_addr":"172.21.15.157:6804/115847","cluster_addr":"172.21.15.157:6805/115847","heartbeat_back_addr":"172.21.15.157:6807/115847","heartbeat_front_addr":"172.21.15.157:6806/115847","state":["exists","up"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":115848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6811","nonce":115848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115848}]},"public_addr":"172.21.15.157:6808/115848","cluster_addr":"172.21.15.157:6809/115848","heartbeat_back_addr":"172.21.15.157:6811/115848","heartbeat_front_addr":"172.21.15.157:6810/115848","state":["exists","up"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6815","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115891}]},"public_addr":"172.21.15.157:6812/115891","cluster_addr":"172.21.15.157:6813/115891","heartbeat_back_addr":"172.21.15.157:6815/115891","heartbeat_front_addr":"172.21.15.157:6814/115891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.324933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.319648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.364624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.391390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.296016+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.336020+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.397884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.386449+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 10 librados: watch_flush enter 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 10 librados: watch_flush exit 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 20 client.4269.objecter shutdown clearing up homeless session... 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 10 client.4269.objecter successfully canceled tick 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 msgr2=0x7f52d406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 0x7f52d406acb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52e000acb0 tx=0x7f52e0008040 comp rx=0 tx=0).stop 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 10 monclient: shutdown 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 msgr2=0x7f52f014dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f52f00a5930 tx=0x7f52ec017000 comp rx=0 tx=0).stop 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 shutdown_connections 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> v2:172.21.15.115:6816/115635 conn(0x7f52d4068800 0x7f52d406acb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f52f01415c0 0x7f52f014e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52f00ae8f0 0x7f52f013e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 --2- 172.21.15.115:0/561190945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52f00aecc0 0x7f52f014dea0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 >> 172.21.15.115:0/561190945 conn(0x7f52f006a460 msgr2=0x7f52f0140960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 shutdown_connections 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 -- 172.21.15.115:0/561190945 wait complete. 2023-09-30T01:03:35.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.741+0000 7f52f873c700 1 librados: shutdown 2023-09-30T01:03:35.754 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T01:03:35.754 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 10 monclient: get_monmap_and_config 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 10 monclient: monmap: 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.870 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:35.871011+0000 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:35.871011+0000 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:35.871 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 1 Processor -- start 2023-09-30T01:03:35.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 1 -- start start 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.870+0000 7f497efe3700 10 monclient: init 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f49780ae8f0 0x7f49780a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-a con 0x7f49780ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49780aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49781415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780a7040 con 0x7f49781415c0 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780a57f0 con 0x7f49780aecc0 2023-09-30T01:03:35.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49780a5480 con 0x7f49780ae8f0 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497efe3700 10 monclient(hunting): authenticate will time out at 1077.321454s 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f49780ae8f0 0x7f49780a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497d580700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f49781415c0 0x7f49780adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46580/0 (socket says 172.21.15.115:46580) 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497d580700 1 -- 172.21.15.115:0/2865880547 learned_addr learned my addr 172.21.15.115:0/2865880547 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): get_auth_request con 0x7f49780ae8f0 auth_method 0 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 1 -- 172.21.15.115:0/2865880547 >> v1:172.21.15.157:6789/0 conn(0x7f49780aecc0 legacy=0x7f49780b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 1 -- 172.21.15.115:0/2865880547 >> v1:172.21.15.115:6789/0 conn(0x7f49781415c0 legacy=0x7f49780adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 1 monclient: found mon.noname-a 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f49777fe700 1 -- 172.21.15.115:0/2865880547 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086927988 0 0) 0x7f49780a7040 con 0x7f49781415c0 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f497cd7f700 1 -- 172.21.15.115:0/2865880547 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780a55c0 con 0x7f49780ae8f0 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.871+0000 7f49777fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:35.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497cd7f700 1 --2- 172.21.15.115:0/2865880547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780ae8f0 0x7f49780a5d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098e0 tx=0x7f496c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d87e8a67593cb8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 1 -- 172.21.15.115:0/2865880547 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c009e80 con 0x7f49780ae8f0 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: dump: 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.884 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 1 -- 172.21.15.115:0/2865880547 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c00b820 con 0x7f49780ae8f0 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 5 monclient: authenticate success, global_id 4272 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 1 -- 172.21.15.115:0/2865880547 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c00bba0 con 0x7f49780ae8f0 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 4 set_mon_vals no callback set 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f49777fe700 10 monclient: dump: 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 10 monclient: shutdown 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 -- 172.21.15.115:0/2865880547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780ae8f0 msgr2=0x7f49780a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 --2- 172.21.15.115:0/2865880547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780ae8f0 0x7f49780a5d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098e0 tx=0x7f496c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 -- 172.21.15.115:0/2865880547 shutdown_connections 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 --2- 172.21.15.115:0/2865880547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780ae8f0 0x7f49780a5d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 -- 172.21.15.115:0/2865880547 >> 172.21.15.115:0/2865880547 conn(0x7f497806a460 msgr2=0x7f49780630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 -- 172.21.15.115:0/2865880547 shutdown_connections 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.872+0000 7f497efe3700 1 -- 172.21.15.115:0/2865880547 wait complete. 2023-09-30T01:03:35.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.873+0000 7f497efe3700 10 monclient: build_initial_monmap 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.873+0000 7f497efe3700 10 monclient: monmap: 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.873+0000 7f497efe3700 1 librados: starting msgr at 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.873+0000 7f497efe3700 1 librados: starting objecter 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.873+0000 7f497efe3700 1 Processor -- start 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 -- start start 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 librados: setting wanted keys 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 librados: calling monclient init 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient: init 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 0x7f4978151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-c con 0x7f49780ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-a con 0x7f49780aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:35.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49781415c0 0x7f49780a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): picked mon.noname-b con 0x7f49781415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49780a8e90 con 0x7f49780aecc0 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49780a7c10 con 0x7f49781415c0 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4978152180 con 0x7f49780ae8f0 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f497efe3700 10 monclient(hunting): authenticate will time out at 1077.324921s 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.874+0000 7f4977fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 0x7f4978151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44030/0 (socket says 172.21.15.115:44030) 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): get_auth_request con 0x7f49780ae8f0 auth_method 0 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 1 -- 172.21.15.115:0/3290717721 learned_addr learned my addr 172.21.15.115:0/3290717721 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): get_auth_request con 0x7f49780aecc0 auth_method 0 2023-09-30T01:03:35.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49781415c0 0x7f49780a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): get_auth_request con 0x7f49781415c0 auth_method 0 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:35.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497d580700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 -- 172.21.15.115:0/3290717721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49781415c0 msgr2=0x7f49780a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49781415c0 0x7f49780a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 -- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 msgr2=0x7f49780a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:35.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient: _finish_hunting 0 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 monclient: found mon.noname-c 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 -- 172.21.15.115:0/3290717721 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c009420 con 0x7f49780ae8f0 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4977fff700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497cd7f700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 0x7f4978151650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f496c015000 tx=0x7f496c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 client.?.objecter ms_handle_connect 0x7f49780ae8f0 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c00ba60 con 0x7f49780ae8f0 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: dump: 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.895 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: _finish_auth 0 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 5 monclient: authenticate success, global_id 4232 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c020070 con 0x7f49780ae8f0 2023-09-30T01:03:35.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c01b6a0 con 0x7f49780ae8f0 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 10 monclient: _renew_subs 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497813d790 con 0x7f49780ae8f0 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4976ffd700 4 set_mon_vals no callback set 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:35.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f4974ff9700 10 monclient: dump: 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:35.898 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:35.899 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:35.899 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:35.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 10 monclient: _renew_subs 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49780a8a90 con 0x7f49780ae8f0 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.875+0000 7f497efe3700 1 librados: init done 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.876+0000 7f497efe3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.876+0000 7f497efe3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.876+0000 7f497efe3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:35.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.876+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4978152340 con 0x7f49780ae8f0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.876+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f496c022030 con 0x7f49780ae8f0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 1 --2- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 0x7f495c06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f496c085b70 con 0x7f49780ae8f0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 10 client.4232.objecter ms_dispatch 0x7f4978001550 osd_map(16..16 src has 1..16) v4 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 3 client.4232.objecter handle_osd_map got epochs [16,16] > 0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 3 client.4232.objecter handle_osd_map decoding full epoch 16 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4977fff700 1 --2- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 0x7f495c06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4977fff700 10 monclient: get_auth_request con 0x7f495c068800 auth_method 0 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 20 client.4232.objecter dump_active .. 0 homeless 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4974ff9700 10 client.4232.objecter ms_handle_connect 0x7f495c068800 2023-09-30T01:03:35.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.877+0000 7f4977fff700 1 --2- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 0x7f495c06acb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49780aab00 tx=0x7f4964007f60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:35.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.881+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4978152340 con 0x7f49780ae8f0 2023-09-30T01:03:35.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.881+0000 7f4974ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:35.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:35.881+0000 7f4974ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.066+0000 7f497efe3700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:36.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.066+0000 7f497efe3700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:36.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.066+0000 7f497efe3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.066+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49780a8c80 con 0x7f49780ae8f0 2023-09-30T01:03:36.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.067+0000 7f4974ff9700 1 -- 172.21.15.115:0/3290717721 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11967 (secure 0 0 0) 0x7f496c053510 con 0x7f49780ae8f0 2023-09-30T01:03:36.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.067+0000 7f4974ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:03:36.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.067+0000 7f4974ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:36.067 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:36.067 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:36.028196+0000","last_up_change":"2023-09-30T01:03:32.012648+0000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:03:32.045422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:03:33.941118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6801","nonce":115935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6803","nonce":115935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115935}]},"public_addr":"172.21.15.115:6800/115935","cluster_addr":"172.21.15.115:6801/115935","heartbeat_back_addr":"172.21.15.115:6803/115935","heartbeat_front_addr":"172.21.15.115:6802/115935","state":["exists","up"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":115950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6811","nonce":115950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115950}]},"public_addr":"172.21.15.115:6808/115950","cluster_addr":"172.21.15.115:6809/115950","heartbeat_back_addr":"172.21.15.115:6811/115950","heartbeat_front_addr":"172.21.15.115:6810/115950","state":["exists","up"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6805","nonce":115951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6807","nonce":115951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115951}]},"public_addr":"172.21.15.115:6804/115951","cluster_addr":"172.21.15.115:6805/115951","heartbeat_back_addr":"172.21.15.115:6807/115951","heartbeat_front_addr":"172.21.15.115:6806/115951","state":["exists","up"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6813","nonce":115952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":115952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115952}]},"public_addr":"172.21.15.115:6812/115952","cluster_addr":"172.21.15.115:6813/115952","heartbeat_back_addr":"172.21.15.115:6815/115952","heartbeat_front_addr":"172.21.15.115:6814/115952","state":["exists","up"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6801","nonce":115846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6803","nonce":115846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115846}]},"public_addr":"172.21.15.157:6800/115846","cluster_addr":"172.21.15.157:6801/115846","heartbeat_back_addr":"172.21.15.157:6803/115846","heartbeat_front_addr":"172.21.15.157:6802/115846","state":["exists","up"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6805","nonce":115847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":115847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115847}]},"public_addr":"172.21.15.157:6804/115847","cluster_addr":"172.21.15.157:6805/115847","heartbeat_back_addr":"172.21.15.157:6807/115847","heartbeat_front_addr":"172.21.15.157:6806/115847","state":["exists","up"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":115848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6811","nonce":115848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115848}]},"public_addr":"172.21.15.157:6808/115848","cluster_addr":"172.21.15.157:6809/115848","heartbeat_back_addr":"172.21.15.157:6811/115848","heartbeat_front_addr":"172.21.15.157:6810/115848","state":["exists","up"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6815","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115891}]},"public_addr":"172.21.15.157:6812/115891","cluster_addr":"172.21.15.157:6813/115891","heartbeat_back_addr":"172.21.15.157:6815/115891","heartbeat_front_addr":"172.21.15.157:6814/115891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.324933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.319648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.364624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.391390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.296016+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.336020+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.397884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.386449+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:03:36.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.068+0000 7f497efe3700 10 librados: watch_flush enter 2023-09-30T01:03:36.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.068+0000 7f497efe3700 10 librados: watch_flush exit 2023-09-30T01:03:36.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.068+0000 7f497efe3700 20 client.4232.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.068+0000 7f497efe3700 10 client.4232.objecter successfully canceled tick 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 msgr2=0x7f495c06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 0x7f495c06acb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f49780aab00 tx=0x7f4964007f60 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 10 monclient: shutdown 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 msgr2=0x7f4978151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 0x7f4978151650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f496c015000 tx=0x7f496c016000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 shutdown_connections 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> v2:172.21.15.115:6816/115635 conn(0x7f495c068800 0x7f495c06acb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49781415c0 0x7f49780a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49780ae8f0 0x7f4978151650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 --2- 172.21.15.115:0/3290717721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49780aecc0 0x7f49780a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 >> 172.21.15.115:0/3290717721 conn(0x7f497806a460 msgr2=0x7f497813fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 shutdown_connections 2023-09-30T01:03:36.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 -- 172.21.15.115:0/3290717721 wait complete. 2023-09-30T01:03:36.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.069+0000 7f497efe3700 1 librados: shutdown 2023-09-30T01:03:36.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T01:03:36.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T01:03:36.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T01:03:36.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T01:03:36.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T01:03:36.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T01:03:36.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T01:03:36.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T01:03:36.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.274+0000 7f60e9987700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.274+0000 7f60e9987700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.274+0000 7f60e9987700 10 monclient: monmap: 2023-09-30T01:03:36.275 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.275 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.275746+0000 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.275746+0000 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.276 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.277 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.277 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.275+0000 7f60e9987700 1 Processor -- start 2023-09-30T01:03:36.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.275+0000 7f60e9987700 1 -- start start 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.275+0000 7f60e9987700 10 monclient: init 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60e4145a30 0x7f60e406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-c con 0x7f60e4145a30 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e41466e0 0x7f60e406f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-b con 0x7f60e41466e0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60e4145e00 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e406d670 con 0x7f60e4145e00 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e4140120 con 0x7f60e4145a30 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40aab40 con 0x7f60e41466e0 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.276+0000 7f60e9987700 10 monclient(hunting): authenticate will time out at 1077.726725s 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60e4145a30 0x7f60e406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60e4145a30 0x7f60e406f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45338/0 (socket says 172.21.15.115:45338) 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 -- 172.21.15.115:0/781889396 learned_addr learned my addr 172.21.15.115:0/781889396 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): get_auth_request con 0x7f60e4145a30 auth_method 0 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 -- 172.21.15.115:0/781889396 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e41466e0 msgr2=0x7f60e406f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/781889396 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e41466e0 0x7f60e406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 -- 172.21.15.115:0/781889396 >> v1:172.21.15.115:6789/0 conn(0x7f60e4145e00 legacy=0x7f60e406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 monclient: found mon.noname-c 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 -- 172.21.15.115:0/781889396 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e406b6b0 con 0x7f60e4145a30 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.277+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/781889396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145a30 0x7f60e406f2e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f60d40099b0 tx=0x7f60d4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e9b0570ec9a2fc6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 1 -- 172.21.15.115:0/781889396 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400b860 con 0x7f60e4145a30 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: dump: 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.282 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.282 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.282 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 1 -- 172.21.15.115:0/781889396 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400b9c0 con 0x7f60e4145a30 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 1 -- 172.21.15.115:0/781889396 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400bd60 con 0x7f60e4145a30 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.281+0000 7f60e1ffb700 10 monclient: dump: 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.284 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.282+0000 7f79a3ca0700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.282+0000 7f79a3ca0700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.282+0000 7f79a3ca0700 10 monclient: monmap: 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.283098+0000 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.283098+0000 2023-09-30T01:03:36.285 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.283+0000 7f60e9987700 5 monclient: authenticate success, global_id 4282 2023-09-30T01:03:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.283+0000 7f60e9987700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.283+0000 7f60e9987700 4 set_mon_vals no callback set 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f56b992c700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f56b992c700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f56b992c700 10 monclient: monmap: 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.286244+0000 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.286244+0000 2023-09-30T01:03:36.287 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.288 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f79a3ca0700 1 Processor -- start 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f79a3ca0700 1 -- start start 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.285+0000 7f79a3ca0700 10 monclient: init 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f799c06e400 0x7f799c06ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-a con 0x7f799c06e400 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f799c13d6d0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f799c13db30 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c06bb30 con 0x7f799c13db30 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c06cbe0 con 0x7f799c13d6d0 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c06d840 con 0x7f799c06e400 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.286+0000 7f79a3ca0700 10 monclient(hunting): authenticate will time out at 1077.736693s 2023-09-30T01:03:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 10 monclient: shutdown 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 -- 172.21.15.115:0/781889396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145a30 msgr2=0x7f60e406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 --2- 172.21.15.115:0/781889396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145a30 0x7f60e406f2e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f60d40099b0 tx=0x7f60d4009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 -- 172.21.15.115:0/781889396 shutdown_connections 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 --2- 172.21.15.115:0/781889396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145a30 0x7f60e406f2e0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f60d40099b0 tx=0x7f60d4009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 --2- 172.21.15.115:0/781889396 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e41466e0 0x7f60e406f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.284+0000 7f60e9987700 1 -- 172.21.15.115:0/781889396 >> 172.21.15.115:0/781889396 conn(0x7f60e40b0540 msgr2=0x7f60e40b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f799c06e400 0x7f799c06ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f799c06e400 0x7f799c06ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44042/0 (socket says 172.21.15.115:44042) 2023-09-30T01:03:36.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 -- 172.21.15.115:0/4073321786 learned_addr learned my addr 172.21.15.115:0/4073321786 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): get_auth_request con 0x7f799c06e400 auth_method 0 2023-09-30T01:03:36.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-09-30T01:03:36.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 -- 172.21.15.115:0/4073321786 >> v1:172.21.15.157:6789/0 conn(0x7f799c13d6d0 legacy=0x7f799c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 -- 172.21.15.115:0/4073321786 >> v1:172.21.15.115:6789/0 conn(0x7f799c13db30 legacy=0x7f799c06c4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 -- 172.21.15.115:0/4073321786 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c0aab40 con 0x7f799c06e400 2023-09-30T01:03:36.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f60e9987700 1 -- 172.21.15.115:0/781889396 shutdown_connections 2023-09-30T01:03:36.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.287+0000 7f79a1a3c700 1 --2- 172.21.15.115:0/4073321786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c06ec40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f798c0089b0 tx=0x7f798c0082d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f319083b94769f31 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 1 -- 172.21.15.115:0/4073321786 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c010070 con 0x7f799c06e400 2023-09-30T01:03:36.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: dump: 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.297 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 1 -- 172.21.15.115:0/4073321786 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c00dab0 con 0x7f799c06e400 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 1 -- 172.21.15.115:0/4073321786 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c00de50 con 0x7f799c06e400 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a0a3a700 10 monclient: dump: 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.298 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a3ca0700 5 monclient: authenticate success, global_id 4281 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a3ca0700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.290+0000 7f79a3ca0700 4 set_mon_vals no callback set 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.291+0000 7f79a3ca0700 10 monclient: shutdown 2023-09-30T01:03:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.291+0000 7f79a3ca0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.291+0000 7f79a3ca0700 1 -- 172.21.15.115:0/4073321786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 msgr2=0x7f799c06ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.291+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/4073321786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c06ec40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f798c0089b0 tx=0x7f798c0082d0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.291+0000 7f60e9987700 1 -- 172.21.15.115:0/781889396 wait complete. 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f60e9987700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f60e9987700 10 monclient: monmap: 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.300 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f79a3ca0700 1 -- 172.21.15.115:0/4073321786 shutdown_connections 2023-09-30T01:03:36.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/4073321786 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c06ec40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f79a3ca0700 1 -- 172.21.15.115:0/4073321786 >> 172.21.15.115:0/4073321786 conn(0x7f799c067bd0 msgr2=0x7f799c0b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f60e9987700 1 librados: starting msgr at 2023-09-30T01:03:36.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f60e9987700 1 librados: starting objecter 2023-09-30T01:03:36.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f56b992c700 1 Processor -- start 2023-09-30T01:03:36.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f56b992c700 1 -- start start 2023-09-30T01:03:36.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f56b992c700 10 monclient: init 2023-09-30T01:03:36.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f60e9987700 1 Processor -- start 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f1e04bc1700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f1e04bc1700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.292+0000 7f1e04bc1700 10 monclient: monmap: 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.293928+0000 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.293928+0000 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.304 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f79a3ca0700 1 -- 172.21.15.115:0/4073321786 shutdown_connections 2023-09-30T01:03:36.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 0x7f56b406eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-b con 0x7f56b40ae4e0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56b40ae8b0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f56b40b1180 0x7f56b406f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-c con 0x7f56b40b1180 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40b0b50 con 0x7f56b40ae8b0 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40b08e0 con 0x7f56b40b1180 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b406d4b0 con 0x7f56b40ae4e0 2023-09-30T01:03:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f56b992c700 10 monclient(hunting): authenticate will time out at 1077.743516s 2023-09-30T01:03:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f1e04bc1700 1 Processor -- start 2023-09-30T01:03:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f1e04bc1700 1 -- start start 2023-09-30T01:03:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f1e04bc1700 10 monclient: init 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b27fc700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f56b40ae8b0 0x7f56b4145ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46604/0 (socket says 172.21.15.115:46604) 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b27fc700 1 -- 172.21.15.115:0/3073332755 learned_addr learned my addr 172.21.15.115:0/3073332755 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f60e9987700 1 -- start start 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f60e9987700 1 librados: setting wanted keys 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f60e9987700 1 librados: calling monclient init 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f60e9987700 10 monclient: init 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145e00 0x7f60e41426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-b con 0x7f60e4145e00 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 0x7f60e41404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-a con 0x7f60e41466e0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e4143830 0x7f60e4143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): picked mon.noname-c con 0x7f60e4143830 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4142da0 con 0x7f60e41466e0 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4143230 con 0x7f60e4145e00 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4141d10 con 0x7f60e4143830 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f60e9987700 10 monclient(hunting): authenticate will time out at 1077.744383s 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f79a3ca0700 1 -- 172.21.15.115:0/4073321786 wait complete. 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f79a3ca0700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.293+0000 7f79a3ca0700 10 monclient: monmap: 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 librados: starting msgr at 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 librados: starting objecter 2023-09-30T01:03:36.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 Processor -- start 2023-09-30T01:03:36.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280597736 0 0) 0x7f56b40b0b50 con 0x7f56b40ae8b0 2023-09-30T01:03:36.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b1ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b1ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b1ffb700 10 monclient(hunting): my global_id is 4284 2023-09-30T01:03:36.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f569c003570 con 0x7f56b40ae8b0 2023-09-30T01:03:36.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f1e04bc1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T01:03:36.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e00145ba0 0x7f1e000ac650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-b con 0x7f1e00145ba0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f1e00146690 0x7f1e00142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-a con 0x7f1e00146690 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e0006bc10 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e00142930 con 0x7f1e0006bc10 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e001400f0 con 0x7f1e00146690 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e000acb90 con 0x7f1e00145ba0 2023-09-30T01:03:36.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1e04bc1700 10 monclient(hunting): authenticate will time out at 1077.745101s 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3073332755 >> v2:172.21.15.157:3300/0 conn(0x7f56b40b1180 0x7f56b406f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): get_auth_request con 0x7f56b40b1180 auth_method 0 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139134318 0 0) 0x7f569c003570 con 0x7f56b40ae8b0 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 msgr2=0x7f56b406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 0x7f56b406eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 >> v2:172.21.15.157:3300/0 conn(0x7f56b40b1180 msgr2=0x7f56b406f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 --2- 172.21.15.115:0/3073332755 >> v2:172.21.15.157:3300/0 conn(0x7f56b40b1180 0x7f56b406f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b406d6b0 con 0x7f56b40ae8b0 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b992c700 5 monclient: authenticate success, global_id 4284 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b992c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f56a4001470 con 0x7f56b40ae8b0 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: dump: 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.318 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 >> v1:172.21.15.115:6789/0 conn(0x7f56b40ae8b0 legacy=0x7f56b4145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 0x7f569c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): picked mon.c con 0x7f56b40ae4e0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f569c004140 0x7f569c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): picked mon.b con 0x7f569c004140 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f569c0052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): picked mon.a con 0x7f569c004e70 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40b08e0 con 0x7f569c004e70 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b406d4b0 con 0x7f569c004140 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c01cd80 con 0x7f56b40ae4e0 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b1ffb700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:03:36.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b992c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 -- start start 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 librados: setting wanted keys 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 1 librados: calling monclient init 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.294+0000 7f79a3ca0700 10 monclient: init 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:36.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c13c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-a con 0x7f799c06e400 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f799c13d6d0 0x7f799c13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-c con 0x7f799c13d6d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 0x7f799c13afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): picked mon.noname-b con 0x7f799c13db30 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c139c20 con 0x7f799c06e400 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c13a0b0 con 0x7f799c13db30 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c14dcb0 con 0x7f799c13d6d0 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a3ca0700 10 monclient(hunting): authenticate will time out at 1077.745568s 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfed9d700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1e0006bc10 0x7f1e000acf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.115:56210/0 (socket says 172.21.15.115:56210) 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfed9d700 1 -- 172.21.15.115:0/1073203142 learned_addr learned my addr 172.21.15.115:0/1073203142 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f569c0052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): get_auth_request con 0x7f569c004e70 auth_method 0 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f56b37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 0x7f799c13afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 0x7f799c13afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45368/0 (socket says 172.21.15.115:45368) 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 1 -- 172.21.15.115:0/960600190 learned_addr learned my addr 172.21.15.115:0/960600190 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 10 monclient(hunting): get_auth_request con 0x7f799c13db30 auth_method 0 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a223d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a1a3c700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c13c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a1a3c700 10 monclient(hunting): get_auth_request con 0x7f799c06e400 auth_method 0 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a1a3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a1a3c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f79a1a3c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 1 --2- 172.21.15.115:0/1073203142 >> v2:172.21.15.115:3300/0 conn(0x7f1e00146690 0x7f1e00142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): get_auth_request con 0x7f1e00146690 auth_method 0 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f1dfdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f60e27fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 0x7f60e41404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.295+0000 7f60e27fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 0x7f60e41404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44056/0 (socket says 172.21.15.115:44056) 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e27fc700 1 -- 172.21.15.115:0/3021478179 learned_addr learned my addr 172.21.15.115:0/3021478179 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e27fc700 10 monclient(hunting): get_auth_request con 0x7f60e41466e0 auth_method 0 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145e00 0x7f60e41426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient(hunting): handle_auth_done global_id 4284 payload 293 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 -- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 msgr2=0x7f569c003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b40ae4e0 0x7f569c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 -- 172.21.15.115:0/3073332755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f569c004140 msgr2=0x7f569c004810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f569c004140 0x7f569c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 monclient: found mon.a 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 -- 172.21.15.115:0/3073332755 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b406d6b0 con 0x7f569c004e70 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a123b700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f799c13d6d0 0x7f799c13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): get_auth_request con 0x7f60e4145e00 auth_method 0 2023-09-30T01:03:36.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f569c0052e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bb90 tx=0x7f56a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67006e36feb61e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7ff01daa6700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7ff01daa6700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7ff01daa6700 10 monclient: monmap: 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.297550+0000 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.297550+0000 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 -- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f799c13d6d0 msgr2=0x7f799c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f799c13d6d0 0x7f799c13aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 -- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 msgr2=0x7f799c13c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f799c06e400 0x7f799c13c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a223d700 1 -- 172.21.15.115:0/960600190 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c008570 con 0x7f799c13db30 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a123b700 10 monclient: get_auth_request con 0x7f799c13d6d0 auth_method 0 2023-09-30T01:03:36.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f79a123b700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f799c13d6d0 0x7f799c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfd59a700 1 -- 172.21.15.115:0/1073203142 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679878250 0 0) 0x7f1e00142930 con 0x7f1e0006bc10 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfd59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfd59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfd59a700 10 monclient(hunting): my global_id is 4288 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfd59a700 1 -- 172.21.15.115:0/1073203142 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec003570 con 0x7f1e0006bc10 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 1 -- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e00145ba0 msgr2=0x7f1e000ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 1 --2- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e00145ba0 0x7f1e000ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 1 -- 172.21.15.115:0/1073203142 >> v1:172.21.15.157:6789/0 conn(0x7f1e0006bc10 legacy=0x7f1e000acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f1dfdd9b700 1 -- 172.21.15.115:0/1073203142 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00145f70 con 0x7f1e00146690 2023-09-30T01:03:36.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a223d700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 0x7f799c13afc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f799800b520 tx=0x7f79980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 client.?.objecter ms_handle_connect 0x7f799c13db30 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 -- 172.21.15.115:0/3021478179 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e4143830 msgr2=0x7f60e4143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f60e4143830 0x7f60e4143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 -- 172.21.15.115:0/3021478179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 msgr2=0x7f60e41404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 0x7f60e41404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.296+0000 7f60e2ffd700 1 -- 172.21.15.115:0/3021478179 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40094f0 con 0x7f60e4145e00 2023-09-30T01:03:36.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f60e27fc700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f60e41466e0 0x7f60e41404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfdd9b700 1 --2- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00146690 0x7f1e00142dc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1df00098e0 tx=0x7f1df00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2d677cc116fad1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 1 -- 172.21.15.115:0/1073203142 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0011630 con 0x7f1e00146690 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a800d5d0 con 0x7f569c004e70 2023-09-30T01:03:36.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: dump: 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.339 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a800d730 con 0x7f569c004e70 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 1 -- 172.21.15.115:0/3073332755 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a800daf0 con 0x7f569c004e70 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: dump: 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.340 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1e04bc1700 5 monclient: authenticate success, global_id 4287 2023-09-30T01:03:36.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1e04bc1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b1ffb700 10 monclient: dump: 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.342 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 1 -- 172.21.15.115:0/1073203142 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df0011790 con 0x7f1e00146690 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 1 -- 172.21.15.115:0/1073203142 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0011b10 con 0x7f1e00146690 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1dfd59a700 10 monclient: dump: 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1e04bc1700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f1e04bc1700 4 set_mon_vals no callback set 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b992c700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f56b992c700 4 set_mon_vals no callback set 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 10 monclient: shutdown 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 1 -- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 msgr2=0x7f569c0052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 1 --2- 172.21.15.115:0/3073332755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f569c0052e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bb90 tx=0x7f56a8008040 comp rx=0 tx=0).stop 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b37fe700 1 -- 172.21.15.115:0/3073332755 reap_dead start 2023-09-30T01:03:36.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 1 -- 172.21.15.115:0/3073332755 shutdown_connections 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f56b992c700 1 -- 172.21.15.115:0/3073332755 >> 172.21.15.115:0/3073332755 conn(0x7f56b4068900 msgr2=0x7f56b4068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 10 monclient: shutdown 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 -- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00146690 msgr2=0x7f1e00142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00146690 0x7f1e00142dc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1df00098e0 tx=0x7f1df00091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 -- 172.21.15.115:0/1073203142 shutdown_connections 2023-09-30T01:03:36.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e00145ba0 0x7f1e000ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f60e2ffd700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145e00 0x7f60e41426f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60d4013000 tx=0x7f60d4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/1073203142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e00146690 0x7f1e00142dc0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1df00098e0 tx=0x7f1df00091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f1e04bc1700 1 -- 172.21.15.115:0/1073203142 >> 172.21.15.115:0/1073203142 conn(0x7f1e000699e0 msgr2=0x7f1e00069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f3f82543700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f3f82543700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f3f82543700 10 monclient: monmap: 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.299856+0000 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.299856+0000 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.348 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.349 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.349 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f56b992c700 1 -- 172.21.15.115:0/3073332755 shutdown_connections 2023-09-30T01:03:36.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f1e04bc1700 1 -- 172.21.15.115:0/1073203142 shutdown_connections 2023-09-30T01:03:36.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f1e04bc1700 1 -- 172.21.15.115:0/1073203142 wait complete. 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799800f510 con 0x7f799c13db30 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: dump: 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.350 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799800f6b0 con 0x7f799c13db30 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799800fa70 con 0x7f799c13db30 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7991ffb700 10 monclient: dump: 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.352 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 5 monclient: authenticate success, global_id 4291 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 10 monclient: _renew_subs 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c13b500 con 0x7f799c13db30 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 10 monclient: _renew_subs 2023-09-30T01:03:36.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c139100 con 0x7f799c13db30 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f79a3ca0700 1 librados: init done 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f7993fff700 4 set_mon_vals no callback set 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 10 monclient: monmap: 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.354 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7ff01daa6700 1 Processor -- start 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 librados: starting msgr at 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 librados: starting objecter 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 Processor -- start 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 -- start start 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 librados: setting wanted keys 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 1 librados: calling monclient init 2023-09-30T01:03:36.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f1e04bc1700 10 monclient: init 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f56b992c700 1 -- 172.21.15.115:0/3073332755 wait complete. 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f56b992c700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f56b992c700 10 monclient: monmap: 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.357 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f56b992c700 1 librados: starting msgr at 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f56b992c700 1 librados: starting objecter 2023-09-30T01:03:36.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f56b992c700 1 Processor -- start 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 0x7f1e000a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-a con 0x7f1e0006bc10 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00145ba0 0x7f1e001551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-b con 0x7f1e00145ba0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e000a65e0 0x7f1e00141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): picked mon.noname-c con 0x7f1e000a65e0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e000a6c10 con 0x7f1e0006bc10 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e000a56b0 con 0x7f1e00145ba0 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e000aa530 con 0x7f1e000a65e0 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f1e04bc1700 10 monclient(hunting): authenticate will time out at 1077.751765s 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 -- start start 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 librados: setting wanted keys 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 librados: calling monclient init 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient: init 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f56b413f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-a con 0x7f569c004e70 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 0x7f56b40a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-b con 0x7f56b40a64c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b413e470 0x7f56b413e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): picked mon.noname-c con 0x7f56b413e470 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b413efb0 con 0x7f569c004e70 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4139a00 con 0x7f56b40a64c0 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40a5dc0 con 0x7f56b413e470 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7f56b992c700 10 monclient(hunting): authenticate will time out at 1077.751935s 2023-09-30T01:03:36.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 client.?.objecter ms_handle_connect 0x7f60e4145e00 2023-09-30T01:03:36.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4011e90 con 0x7f60e4145e00 2023-09-30T01:03:36.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f01f97d1700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f01f97d1700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.297+0000 7f01f97d1700 10 monclient: monmap: 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.298267+0000 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.298267+0000 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.365 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f01f97d1700 1 Processor -- start 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f01f97d1700 1 -- start start 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.298+0000 7f01f97d1700 10 monclient: init 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f01f40b0e10 0x7f01f406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-c con 0x7f01f40b0e10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01f406c090 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f01f40ab5c0 0x7f01f41557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-a con 0x7f01f40ab5c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f406e220 con 0x7f01f406c090 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f406d300 con 0x7f01f40ab5c0 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f406d5e0 con 0x7f01f40b0e10 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f97d1700 10 monclient(hunting): authenticate will time out at 1077.749303s 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f01f40b0e10 0x7f01f406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f01f40b0e10 0x7f01f406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45388/0 (socket says 172.21.15.115:45388) 2023-09-30T01:03:36.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 1 -- 172.21.15.115:0/2521390939 learned_addr learned my addr 172.21.15.115:0/2521390939 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 10 monclient(hunting): get_auth_request con 0x7f01f40b0e10 auth_method 0 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: dump: 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.369 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d4024430 con 0x7f60e4145e00 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d40247f0 con 0x7f60e4145e00 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60caffd700 10 monclient: dump: 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.370 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 5 monclient: authenticate success, global_id 4294 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 10 monclient: _renew_subs 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4141ff0 con 0x7f60e4145e00 2023-09-30T01:03:36.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 10 monclient: _renew_subs 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e4144210 con 0x7f60e4145e00 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e9987700 1 librados: init done 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 0x7f56b40a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 0x7f56b40a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45406/0 (socket says 172.21.15.115:45406) 2023-09-30T01:03:36.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 -- 172.21.15.115:0/3584668813 learned_addr learned my addr 172.21.15.115:0/3584668813 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): get_auth_request con 0x7f56b40a64c0 auth_method 0 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f60e0ff9700 4 set_mon_vals no callback set 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 -- 172.21.15.115:0/3584668813 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b413e470 msgr2=0x7f56b413e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 --2- 172.21.15.115:0/3584668813 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f56b413e470 0x7f56b413e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 -- 172.21.15.115:0/3584668813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 msgr2=0x7f56b413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 --2- 172.21.15.115:0/3584668813 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f569c004e70 0x7f56b413f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 -- 172.21.15.115:0/3584668813 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b413ce10 con 0x7f56b40a64c0 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f56b27fc700 1 --2- 172.21.15.115:0/3584668813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 0x7f56b40a6910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56a4001620 tx=0x7f56a4001d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f569affd700 10 client.?.objecter ms_handle_connect 0x7f56b40a64c0 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f569affd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4007360 con 0x7f56b40a64c0 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: dump: 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.375 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a4007500 con 0x7f56b40a64c0 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4004a80 con 0x7f56b40a64c0 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f569affd700 10 monclient: dump: 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.377 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 5 monclient: authenticate success, global_id 4306 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 10 monclient: _renew_subs 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b413b930 con 0x7f56b40a64c0 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 10 monclient: _renew_subs 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b413bdd0 con 0x7f56b40a64c0 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f56b992c700 1 librados: init done 2023-09-30T01:03:36.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 10 client.4306.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 15 client.4306.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 20 client.4306.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56b4072960 con 0x7f56b40a64c0 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.299+0000 7f01f37fe700 1 --2- 172.21.15.115:0/2521390939 >> v2:172.21.15.115:3300/0 conn(0x7f01f40ab5c0 0x7f01f41557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f37fe700 10 monclient(hunting): get_auth_request con 0x7f01f40ab5c0 auth_method 0 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7f01f1ffb700 1 -- 172.21.15.115:0/2521390939 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452460115 0 0) 0x7f01f406e220 con 0x7f01f406c090 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f1ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f1ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f1ffb700 10 monclient(hunting): my global_id is 4297 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f1ffb700 1 -- 172.21.15.115:0/2521390939 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01dc007980 con 0x7f01f406c090 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 1 -- 172.21.15.115:0/2521390939 >> v2:172.21.15.115:3300/0 conn(0x7f01f40ab5c0 msgr2=0x7f01f41557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 1 --2- 172.21.15.115:0/2521390939 >> v2:172.21.15.115:3300/0 conn(0x7f01f40ab5c0 0x7f01f41557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 1 -- 172.21.15.115:0/2521390939 >> v1:172.21.15.157:6789/0 conn(0x7f01f406c090 legacy=0x7f01f40aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 1 monclient: found mon.noname-c 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7f01f2ffd700 1 -- 172.21.15.115:0/2521390939 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f406d8c0 con 0x7f01f40b0e10 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f01f2ffd700 1 --2- 172.21.15.115:0/2521390939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f40b0e10 0x7f01f406dce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01e40098b0 tx=0x7f01e4007100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee9b39fa678db8cc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f799801a400 con 0x7f799c13db30 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.115:6816/115635 conn(0x7f7980068950 0x7f798006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7998083890 con 0x7f799c13db30 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 10 client.4291.objecter ms_dispatch 0x7f799c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 3 client.4291.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 3 client.4291.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f7991ffb700 20 client.4291.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f79a1a3c700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.115:6816/115635 conn(0x7f7980068950 0x7f798006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f79a1a3c700 10 monclient: get_auth_request con 0x7f7980068950 auth_method 0 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f79a1a3c700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.115:6816/115635 conn(0x7f7980068950 0x7f798006ae00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f798c0089b0 tx=0x7f798c01b000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f1dfe59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 0x7f1e000a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 0x7f1e000a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44084/0 (socket says 172.21.15.115:44084) 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 -- 172.21.15.115:0/720649932 learned_addr learned my addr 172.21.15.115:0/720649932 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): get_auth_request con 0x7f1e0006bc10 auth_method 0 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 10 client.4294.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 15 client.4294.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 20 client.4294.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60c40028c0 con 0x7f60e4145e00 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f60d401a420 con 0x7f60e4145e00 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 -- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e000a65e0 msgr2=0x7f1e00141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfed9d700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e000a65e0 0x7f1e00141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e000a65e0 0x7f1e00141870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 -- 172.21.15.115:0/720649932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00145ba0 msgr2=0x7f1e001551a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00145ba0 0x7f1e001551a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfe59c700 1 -- 172.21.15.115:0/720649932 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0009420 con 0x7f1e0006bc10 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfed9d700 10 monclient: get_auth_request con 0x7f1e000a65e0 auth_method 0 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f1dfed9d700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1e000a65e0 0x7f1e00141870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:36.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f3f82543700 1 Processor -- start 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f56b0ff9700 4 set_mon_vals no callback set 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 1 -- 172.21.15.115:0/2521390939 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40092e0 con 0x7f01f40b0e10 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: dump: 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.386 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 1 -- 172.21.15.115:0/2521390939 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e4015ab0 con 0x7f01f40b0e10 2023-09-30T01:03:36.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 1 -- 172.21.15.115:0/2521390939 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4015e30 con 0x7f01f40b0e10 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f01f1ffb700 10 monclient: dump: 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.388 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 5 monclient: authenticate success, global_id 4303 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 4 set_mon_vals no callback set 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f56a400a720 con 0x7f56b40a64c0 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6816/115635 conn(0x7f56a0068880 0x7f56a006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f56a4083830 con 0x7f56b40a64c0 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 10 client.4306.objecter ms_dispatch 0x7f56b4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 3 client.4306.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 3 client.4306.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 10 client.4306.objecter checking command 1 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6800/115935 conn(0x7f56a006e7f0 0x7f56a0070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter _get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 15 client.4306.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter _get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 15 client.4306.objecter _session_command_op_assign 0 1 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 4 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 10 client.4306.objecter _send_command 1 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 --> v2:172.21.15.115:6800/115935 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56a0071270 con 0x7f56a006e7f0 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 20 client.4306.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a4085d80 con 0x7f56b40a64c0 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f569affd700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f56b2ffd700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6816/115635 conn(0x7f56a0068880 0x7f56a006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f56b2ffd700 10 monclient: get_auth_request con 0x7f56a0068880 auth_method 0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6800/115935 conn(0x7f56a006e7f0 0x7f56a0070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f56b37fe700 10 monclient: get_auth_request con 0x7f56a006e7f0 auth_method 0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f56b37fe700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6800/115935 conn(0x7f56a006e7f0 0x7f56a0070c00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f569affd700 10 client.4306.objecter ms_handle_connect 0x7f56a006e7f0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f56b2ffd700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6816/115635 conn(0x7f56a0068880 0x7f56a006ad30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00acd0 tx=0x7f56ac009350 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 10 client.4306.objecter ms_handle_connect 0x7f56a0068880 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== osd.0 v2:172.21.15.115:6800/115935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f56a0071270 con 0x7f56a006e7f0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 10 client.4306.objecter ms_dispatch 0x7f56b4001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 10 client.4306.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 4 2023-09-30T01:03:36.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f569affd700 15 client.4306.objecter _session_command_op_remove 0 1 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 10 monclient: shutdown 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 1 -- 172.21.15.115:0/2521390939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f40b0e10 msgr2=0x7f01f406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f01f97d1700 1 --2- 172.21.15.115:0/2521390939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f40b0e10 0x7f01f406dce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f01e40098b0 tx=0x7f01e4007100 comp rx=0 tx=0).stop 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 -- 172.21.15.115:0/2521390939 shutdown_connections 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 --2- 172.21.15.115:0/2521390939 >> v2:172.21.15.115:3300/0 conn(0x7f01f40ab5c0 0x7f01f41557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 --2- 172.21.15.115:0/2521390939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f40b0e10 0x7f01f406dce0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 -- 172.21.15.115:0/2521390939 >> 172.21.15.115:0/2521390939 conn(0x7f01f4067bd0 msgr2=0x7f01f40b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7ff01daa6700 1 -- start start 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.300+0000 7ff01daa6700 10 monclient: init 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff0180ad290 0x7ff01806d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-a con 0x7ff0180ad290 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0180ad660 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01806e9d0 0x7ff01813d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-b con 0x7ff01806e9d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01806de30 con 0x7ff0180ad660 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01806ce10 con 0x7ff0180ad290 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01806d080 con 0x7ff01806e9d0 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff01daa6700 10 monclient(hunting): authenticate will time out at 1077.751604s 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff016ffd700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff0180ad660 0x7ff01806e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.115:56214/0 (socket says 172.21.15.115:56214) 2023-09-30T01:03:36.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.301+0000 7ff016ffd700 1 -- 172.21.15.115:0/872044853 learned_addr learned my addr 172.21.15.115:0/872044853 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137262812 0 0) 0x7ff01806de30 con 0x7ff0180ad660 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): my global_id is 4300 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff000003570 con 0x7ff0180ad660 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 467033888 0 0) 0x7ff000003570 con 0x7ff0180ad660 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01806e9d0 msgr2=0x7ff01813d810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01806e9d0 0x7ff01813d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 >> v2:172.21.15.115:3300/0 conn(0x7ff0180ad290 msgr2=0x7ff01806d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 --2- 172.21.15.115:0/872044853 >> v2:172.21.15.115:3300/0 conn(0x7ff0180ad290 0x7ff01806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 monclient: found mon.noname-c-legacy 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.157:6789/0 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01806ee30 con 0x7ff0180ad660 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7ff00c001470 con 0x7ff0180ad660 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: dump: 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.397 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] but i'm connected to v1:172.21.15.157:6789/0, reconnecting 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 >> v1:172.21.15.157:6789/0 conn(0x7ff0180ad660 legacy=0x7ff01806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff000003ae0 0x7ff000003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): picked mon.c con 0x7ff000003ae0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): picked mon.b con 0x7ff000004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff000005310 0x7ff00002da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): picked mon.a con 0x7ff000005310 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01806ce10 con 0x7ff000005310 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01806d080 con 0x7ff000004810 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00002e330 con 0x7ff000003ae0 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.302+0000 7ff0167fc700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7ff016ffd700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7ff016ffd700 10 monclient(hunting): get_auth_request con 0x7ff000004810 auth_method 0 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7ff016ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7ff016ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.303+0000 7ff016ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:36.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient(hunting): handle_auth_done global_id 4300 payload 293 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 -- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff000003ae0 msgr2=0x7ff000003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff000003ae0 0x7ff000003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 -- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff000005310 msgr2=0x7ff00002da70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff000005310 0x7ff00002da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 monclient: found mon.b 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 -- 172.21.15.115:0/872044853 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01806ee30 con 0x7ff000004810 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff016ffd700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff000004c80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c410 tx=0x7ff00c007940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61577e78e903e46c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00ccc0 con 0x7ff000004810 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 10 monclient: dump: 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c00ce40 con 0x7ff000004810 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7ff0167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7ff0167fc700 1 -- 172.21.15.115:0/872044853 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c005b70 con 0x7ff000004810 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7ff0167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7ff0167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7ff0167fc700 10 monclient: dump: 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.403 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7ff01daa6700 5 monclient: authenticate success, global_id 4300 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7ff01daa6700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7ff01daa6700 4 set_mon_vals no callback set 2023-09-30T01:03:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7ff01daa6700 10 monclient: shutdown 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7ff01daa6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7ff01daa6700 1 -- 172.21.15.115:0/872044853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 msgr2=0x7ff000004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 10 client.4291.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.157:6804/115847 conn(0x7f797c002f70 0x7f797c005420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 20 client.4291.objecter _get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 20 client.4291.objecter _get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 20 client.4291.objecter get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_assign 5 1 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 4 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 10 client.4291.objecter _send_command 1 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 --> v2:172.21.15.157:6804/115847 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f797c008420 con 0x7f797c002f70 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f79a123b700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.157:6804/115847 conn(0x7f797c002f70 0x7f797c005420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f79a123b700 10 monclient: get_auth_request con 0x7f797c002f70 auth_method 0 2023-09-30T01:03:36.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f79a123b700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.157:6804/115847 conn(0x7f797c002f70 0x7f797c005420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f7991ffb700 10 client.4291.objecter ms_handle_connect 0x7f7980068950 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f7991ffb700 10 client.4291.objecter ms_handle_connect 0x7f797c002f70 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1dfe59c700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 0x7f1e000a5fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1de800b840 tx=0x7f1de80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 client.?.objecter ms_handle_connect 0x7f1e0006bc10 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de800f510 con 0x7f1e0006bc10 2023-09-30T01:03:36.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: dump: 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.408 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de800f690 con 0x7f1e0006bc10 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de800fa50 con 0x7f1e0006bc10 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f1df67fc700 10 monclient: dump: 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.409 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1df7fff700 4 set_mon_vals no callback set 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 5 monclient: authenticate success, global_id 4299 2023-09-30T01:03:36.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 10 monclient: _renew_subs 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e00145760 con 0x7f1e0006bc10 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 10 monclient: _renew_subs 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e00141db0 con 0x7f1e0006bc10 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f1e04bc1700 1 librados: init done 2023-09-30T01:03:36.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.115:6816/115635 conn(0x7f60d0068950 0x7f60d006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f60d4085200 con 0x7f60e4145e00 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 10 client.4294.objecter ms_dispatch 0x7f60e4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 3 client.4294.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.304+0000 7f60caffd700 3 client.4294.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 10 client.4294.objecter checking command 1 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.157:6812/115891 conn(0x7f60d006e750 0x7f60d0070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter _get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 15 client.4294.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter _get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 15 client.4294.objecter _session_command_op_assign 7 1 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 4 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 10 client.4294.objecter _send_command 1 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 --> v2:172.21.15.157:6812/115891 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60d00711d0 con 0x7f60d006e750 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 20 client.4294.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60d4087750 con 0x7f60e4145e00 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60caffd700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60e27fc700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.115:6816/115635 conn(0x7f60d0068950 0x7f60d006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60e27fc700 10 monclient: get_auth_request con 0x7f60d0068950 auth_method 0 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.305+0000 7f60e27fc700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.115:6816/115635 conn(0x7f60d0068950 0x7f60d006ae00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f60e4145210 tx=0x7f60d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f60e37fe700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.157:6812/115891 conn(0x7f60d006e750 0x7f60d0070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f60e37fe700 10 monclient: get_auth_request con 0x7f60d006e750 auth_method 0 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 10 client.4299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 15 client.4299.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 20 client.4299.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e00072960 con 0x7f1e0006bc10 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== osd.5 v2:172.21.15.157:6804/115847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f797c008420 con 0x7f797c002f70 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f7991ffb700 10 client.4291.objecter ms_dispatch 0x7f799c001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f7991ffb700 10 client.4291.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f7991ffb700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 4 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f7991ffb700 15 client.4291.objecter _session_command_op_remove 5 1 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7ff01daa6700 1 --2- 172.21.15.115:0/872044853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff000004c80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c410 tx=0x7ff00c007940 comp rx=0 tx=0).stop 2023-09-30T01:03:36.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7ff017fff700 1 -- 172.21.15.115:0/872044853 reap_dead start 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7ff01daa6700 1 -- 172.21.15.115:0/872044853 shutdown_connections 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7ff01daa6700 1 -- 172.21.15.115:0/872044853 >> 172.21.15.115:0/872044853 conn(0x7ff0180699e0 msgr2=0x7ff018069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f3f82543700 1 -- start start 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f3f82543700 10 monclient: init 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f7c0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3f7c0a5c80 0x7f3f7c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c0a5c80 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0a7040 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a5480 con 0x7f3f7c0a5c80 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a5760 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f3f82543700 10 monclient(hunting): authenticate will time out at 1077.759959s 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59004/0 (socket says 172.21.15.115:59004) 2023-09-30T01:03:36.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 -- 172.21.15.115:0/117420405 learned_addr learned my addr 172.21.15.115:0/117420405 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): get_auth_request con 0x7f3f7c0ae8f0 auth_method 0 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 -- 172.21.15.115:0/117420405 >> v2:172.21.15.115:3300/0 conn(0x7f3f7c0a5c80 msgr2=0x7f3f7c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 --2- 172.21.15.115:0/117420405 >> v2:172.21.15.115:3300/0 conn(0x7f3f7c0a5c80 0x7f3f7c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 -- 172.21.15.115:0/117420405 >> v1:172.21.15.157:6789/0 conn(0x7f3f7c0aecc0 legacy=0x7f3f7c0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f3f7bfff700 1 -- 172.21.15.115:0/117420405 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c141b00 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f1de8024430 con 0x7f1e0006bc10 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.115:6816/115635 conn(0x7f1de0068880 0x7f1de006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1de8084460 con 0x7f1e0006bc10 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 10 client.4299.objecter ms_dispatch 0x7f1e00001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 3 client.4299.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 3 client.4299.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 10 client.4299.objecter checking command 1 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.157:6808/115848 conn(0x7f1de006e7f0 0x7f1de0070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter _get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 15 client.4299.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter _get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 15 client.4299.objecter _session_command_op_assign 6 1 2023-09-30T01:03:36.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 4 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 10 client.4299.objecter _send_command 1 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 --> v2:172.21.15.157:6808/115848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1de0071270 con 0x7f1de006e7f0 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 20 client.4299.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de8084740 con 0x7f1e0006bc10 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f1df67fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f60e37fe700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.157:6812/115891 conn(0x7f60d006e750 0x7f60d0070b60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 -- 172.21.15.115:0/2521390939 shutdown_connections 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.306+0000 7f01f97d1700 1 -- 172.21.15.115:0/2521390939 wait complete. 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f01f97d1700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f01f97d1700 10 monclient: monmap: 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.423 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f01f97d1700 1 librados: starting msgr at 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f01f97d1700 1 librados: starting objecter 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.307+0000 7f01f97d1700 1 Processor -- start 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f01f97d1700 1 -- start start 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f01f97d1700 1 librados: setting wanted keys 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f01f97d1700 1 librados: calling monclient init 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.308+0000 7f01f97d1700 10 monclient: init 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f406c090 0x7f01f407cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-b con 0x7f01f406c090 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01f40ab5c0 0x7f01f407bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-a con 0x7f01f40ab5c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 0x7f01f4151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): picked mon.noname-c con 0x7f01f40b0e10 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f407e100 con 0x7f01f40ab5c0 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f407e300 con 0x7f01f406c090 2023-09-30T01:03:36.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40874b0 con 0x7f01f40b0e10 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.309+0000 7f01f97d1700 10 monclient(hunting): authenticate will time out at 1077.759915s 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 0x7f01f4151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 0x7f01f4151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59018/0 (socket says 172.21.15.115:59018) 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 1 -- 172.21.15.115:0/1188621722 learned_addr learned my addr 172.21.15.115:0/1188621722 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): get_auth_request con 0x7f01f40b0e10 auth_method 0 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f2ffd700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f406c090 0x7f01f407cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f1dfed9d700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.157:6808/115848 conn(0x7f1de006e7f0 0x7f1de0070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f1dfed9d700 10 monclient: get_auth_request con 0x7f1de006e7f0 auth_method 0 2023-09-30T01:03:36.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f2ffd700 10 monclient(hunting): get_auth_request con 0x7f01f406c090 auth_method 0 2023-09-30T01:03:36.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f27fc700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01f40ab5c0 0x7f01f407bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f27fc700 10 monclient(hunting): get_auth_request con 0x7f01f40ab5c0 auth_method 0 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.310+0000 7f01f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 -- 172.21.15.115:0/1188621722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f406c090 msgr2=0x7f01f407cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f406c090 0x7f01f407cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 -- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01f40ab5c0 msgr2=0x7f01f407bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01f40ab5c0 0x7f01f407bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 monclient: found mon.noname-c 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f37fe700 1 -- 172.21.15.115:0/1188621722 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e400d040 con 0x7f01f40b0e10 2023-09-30T01:03:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f01f27fc700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01f40ab5c0 0x7f01f407bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 10 client.4294.objecter ms_handle_connect 0x7f60d0068950 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 10 client.4294.objecter ms_handle_connect 0x7f60d006e750 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== osd.7 v2:172.21.15.157:6812/115891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f60d00711d0 con 0x7f60d006e750 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 10 client.4294.objecter ms_dispatch 0x7f60e4001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 10 client.4294.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 4 2023-09-30T01:03:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f60caffd700 15 client.4294.objecter _session_command_op_remove 7 1 2023-09-30T01:03:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f1dfdd9b700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.115:6816/115635 conn(0x7f1de0068880 0x7f1de006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f3f7bfff700 1 --2- 172.21.15.115:0/117420405 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0099b0 tx=0x7f3f6c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf0b9457c6454e4e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f1dfdd9b700 10 monclient: get_auth_request con 0x7f1de0068880 auth_method 0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.311+0000 7f1dfed9d700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.157:6808/115848 conn(0x7f1de006e7f0 0x7f1de0070c00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1dfdd9b700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.115:6816/115635 conn(0x7f1de0068880 0x7f1de006ad30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1e000b08d0 tx=0x7f1df0016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 10 client.4299.objecter ms_handle_connect 0x7f1de006e7f0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 10 client.4299.objecter ms_handle_connect 0x7f1de0068880 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 1 -- 172.21.15.115:0/117420405 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c014b70 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: dump: 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.435 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 1 -- 172.21.15.115:0/117420405 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c014cd0 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== osd.6 v2:172.21.15.157:6808/115848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f1de0071270 con 0x7f1de006e7f0 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 10 client.4299.objecter ms_dispatch 0x7f1e00001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 1 -- 172.21.15.115:0/117420405 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c00b7a0 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f7affd700 10 monclient: dump: 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.437 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 10 client.4299.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 4 2023-09-30T01:03:36.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f1df67fc700 15 client.4299.objecter _session_command_op_remove 6 1 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f82543700 5 monclient: authenticate success, global_id 4238 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f82543700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f3f82543700 4 set_mon_vals no callback set 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 10 monclient: shutdown 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 -- 172.21.15.115:0/117420405 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 msgr2=0x7f3f7c1415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 --2- 172.21.15.115:0/117420405 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0099b0 tx=0x7f3f6c009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 -- 172.21.15.115:0/117420405 shutdown_connections 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 --2- 172.21.15.115:0/117420405 >> v2:172.21.15.115:3300/0 conn(0x7f3f7c0a5c80 0x7f3f7c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 --2- 172.21.15.115:0/117420405 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0ae8f0 0x7f3f7c1415c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7f3f82543700 1 -- 172.21.15.115:0/117420405 >> 172.21.15.115:0/117420405 conn(0x7f3f7c06a490 msgr2=0x7f3f7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.313+0000 7ff01daa6700 1 -- 172.21.15.115:0/872044853 shutdown_connections 2023-09-30T01:03:36.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.316+0000 7f098e02f700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.316+0000 7f098e02f700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.316+0000 7f098e02f700 10 monclient: monmap: 2023-09-30T01:03:36.441 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.317840+0000 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.317840+0000 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.442 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.317+0000 7f098e02f700 1 Processor -- start 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.317+0000 7f098e02f700 1 -- start start 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.317+0000 7f098e02f700 10 monclient: init 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09880ae8f0 0x7f09880a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-c con 0x7f09880ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f09880b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-b con 0x7f09880aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09881415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880a7040 con 0x7f09881415c0 2023-09-30T01:03:36.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09880b19b0 con 0x7f09880ae8f0 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09880a57f0 con 0x7f09880aecc0 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f098e02f700 10 monclient(hunting): authenticate will time out at 1077.768464s 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0987fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f09881415c0 0x7f09880adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46610/0 (socket says 172.21.15.115:46610) 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0987fff700 1 -- 172.21.15.115:0/2105195639 learned_addr learned my addr 172.21.15.115:0/2105195639 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 1 --2- 172.21.15.115:0/2105195639 >> v2:172.21.15.157:3300/0 conn(0x7f09880ae8f0 0x7f09880a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): get_auth_request con 0x7f09880ae8f0 auth_method 0 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0986ffd700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f09880b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0986ffd700 10 monclient(hunting): get_auth_request con 0x7f09880aecc0 auth_method 0 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0986ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0986ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f0986ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 -- 172.21.15.115:0/2105195639 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 msgr2=0x7f09880b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f09880b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 -- 172.21.15.115:0/2105195639 >> v1:172.21.15.115:6789/0 conn(0x7f09881415c0 legacy=0x7f09880adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 monclient: found mon.noname-c 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 -- 172.21.15.115:0/2105195639 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880a5480 con 0x7f09880ae8f0 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09877fe700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09880ae8f0 0x7f09880a5d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09700098e0 tx=0x7f09700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41b3f0dc12457d96 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 1 -- 172.21.15.115:0/2105195639 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970009e80 con 0x7f09880ae8f0 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: dump: 2023-09-30T01:03:36.446 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f098e02f700 5 monclient: authenticate success, global_id 4315 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f098e02f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 1 -- 172.21.15.115:0/2105195639 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097000b860 con 0x7f09880ae8f0 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 1 -- 172.21.15.115:0/2105195639 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097000bbe0 con 0x7f09880ae8f0 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f098e02f700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f098e02f700 4 set_mon_vals no callback set 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f09867fc700 10 monclient: dump: 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.448 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.449 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01f37fe700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 0x7f01f4151310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009fd0 tx=0x7f01ec00b1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 client.?.objecter ms_handle_connect 0x7f01f40b0e10 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec010070 con 0x7f01f40b0e10 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: dump: 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.451 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec008a90 con 0x7f01f40b0e10 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec008e90 con 0x7f01f40b0e10 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01daffd700 10 monclient: dump: 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.312+0000 7f01f17fa700 4 set_mon_vals no callback set 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 5 monclient: authenticate success, global_id 4235 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 monclient: _renew_subs 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f407c1e0 con 0x7f01f40b0e10 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 monclient: _renew_subs 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f4151850 con 0x7f01f40b0e10 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 1 librados: init done 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 client.4235.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 15 client.4235.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 20 client.4235.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.315+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01f4072960 con 0x7f01f40b0e10 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.317+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f01ec014030 con 0x7f01f40b0e10 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.317+0000 7f01daffd700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6816/115635 conn(0x7f01e0079960 0x7f01e007be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f01ec084b90 con 0x7f01f40b0e10 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 10 client.4235.objecter ms_dispatch 0x7f01f4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 3 client.4235.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 3 client.4235.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 10 client.4235.objecter checking command 1 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6808/115950 conn(0x7f01e007f760 0x7f01e0081b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter _get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 15 client.4235.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter _get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 15 client.4235.objecter _session_command_op_assign 1 1 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 4 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 10 client.4235.objecter _send_command 1 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 --> v2:172.21.15.115:6808/115950 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01e00821e0 con 0x7f01e007f760 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 20 client.4235.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01ec084e70 con 0x7f01f40b0e10 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.318+0000 7f01daffd700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f01f2ffd700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6816/115635 conn(0x7f01e0079960 0x7f01e007be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f01f2ffd700 10 monclient: get_auth_request con 0x7f01e0079960 auth_method 0 2023-09-30T01:03:36.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f01f2ffd700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6816/115635 conn(0x7f01e0079960 0x7f01e007be10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01e40098b0 tx=0x7f01e401d000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.319+0000 7f01daffd700 10 client.4235.objecter ms_handle_connect 0x7f01e0079960 2023-09-30T01:03:36.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 10 monclient: shutdown 2023-09-30T01:03:36.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 -- 172.21.15.115:0/2105195639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09880ae8f0 msgr2=0x7f09880a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09880ae8f0 0x7f09880a5d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09700098e0 tx=0x7f09700091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 -- 172.21.15.115:0/2105195639 shutdown_connections 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09880ae8f0 0x7f09880a5d10 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09700098e0 tx=0x7f09700091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 --2- 172.21.15.115:0/2105195639 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f09880b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 -- 172.21.15.115:0/2105195639 >> 172.21.15.115:0/2105195639 conn(0x7f098806a460 msgr2=0x7f09880630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 -- 172.21.15.115:0/2105195639 shutdown_connections 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 1 -- 172.21.15.115:0/2105195639 wait complete. 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.320+0000 7f098e02f700 10 monclient: monmap: 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.459 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 librados: starting msgr at 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 librados: starting objecter 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 Processor -- start 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 -- start start 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 librados: setting wanted keys 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 1 librados: calling monclient init 2023-09-30T01:03:36.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.321+0000 7f098e02f700 10 monclient: init 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f098806e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-c con 0x7f09880aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09881415c0 0x7f098813e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-b con 0x7f09881415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 0x7f098813e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): picked mon.noname-a con 0x7f098813d7c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098806dd40 con 0x7f098813d7c0 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098806e020 con 0x7f09881415c0 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098806ad80 con 0x7f09880aecc0 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f098e02f700 10 monclient(hunting): authenticate will time out at 1077.772382s 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 0x7f098813e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 0x7f098813e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44118/0 (socket says 172.21.15.115:44118) 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 1 -- 172.21.15.115:0/3294498983 learned_addr learned my addr 172.21.15.115:0/3294498983 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): get_auth_request con 0x7f098813d7c0 auth_method 0 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 1 --2- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f098806e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): get_auth_request con 0x7f09880aecc0 auth_method 0 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f09877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.322+0000 7f0987fff700 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-09-30T01:03:36.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 -- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 msgr2=0x7f098806e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 --2- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09880aecc0 0x7f098806e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 -- 172.21.15.115:0/3294498983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09881415c0 msgr2=0x7f098813e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 --2- 172.21.15.115:0/3294498983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09881415c0 0x7f098813e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 -- 172.21.15.115:0/3294498983 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009420 con 0x7f098813d7c0 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f0987fff700 1 --2- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 0x7f098813e740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f098000cbc0 tx=0x7f09800019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f097f7fe700 10 client.?.objecter ms_handle_connect 0x7f098813d7c0 2023-09-30T01:03:36.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f097f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f3f82543700 1 -- 172.21.15.115:0/117420405 shutdown_connections 2023-09-30T01:03:36.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.323+0000 7f3f82543700 1 -- 172.21.15.115:0/117420405 wait complete. 2023-09-30T01:03:36.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098000a890 con 0x7f098813d7c0 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: dump: 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098000a9f0 con 0x7f098813d7c0 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098000ad70 con 0x7f098813d7c0 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f097f7fe700 10 monclient: dump: 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 5 monclient: authenticate success, global_id 4305 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 10 monclient: _renew_subs 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098806e980 con 0x7f098813d7c0 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 10 monclient: _renew_subs 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098813ee40 con 0x7f098813d7c0 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f098e02f700 1 librados: init done 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f3f82543700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f3f82543700 10 monclient: monmap: 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f3f82543700 1 librados: starting msgr at 2023-09-30T01:03:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f3f82543700 1 librados: starting objecter 2023-09-30T01:03:36.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.324+0000 7f3f82543700 1 Processor -- start 2023-09-30T01:03:36.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7ff01daa6700 1 -- 172.21.15.115:0/872044853 wait complete. 2023-09-30T01:03:36.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 -- start start 2023-09-30T01:03:36.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 librados: setting wanted keys 2023-09-30T01:03:36.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 librados: calling monclient init 2023-09-30T01:03:36.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient: init 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0a5c80 0x7f3f7c0a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7c0ae8f0 0x7f3f7c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f7c0aecc0 0x7f3f7c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c14d510 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a8ea0 con 0x7f3f7c0ae8f0 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a7f60 con 0x7f3f7c0a5c80 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.325+0000 7f3f82543700 10 monclient(hunting): authenticate will time out at 1077.775966s 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0a5c80 0x7f3f7c0a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0a5c80 0x7f3f7c0a7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59052/0 (socket says 172.21.15.115:59052) 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 1 -- 172.21.15.115:0/374241297 learned_addr learned my addr 172.21.15.115:0/374241297 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): get_auth_request con 0x7f3f7c0a5c80 auth_method 0 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f7c0aecc0 0x7f3f7c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): get_auth_request con 0x7f3f7c0aecc0 auth_method 0 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7b7fe700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7c0ae8f0 0x7f3f7c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7b7fe700 10 monclient(hunting): get_auth_request con 0x7f3f7c0ae8f0 auth_method 0 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 -- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0a5c80 msgr2=0x7f3f7c0a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3f7c0a5c80 0x7f3f7c0a7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 -- 172.21.15.115:0/374241297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7c0ae8f0 msgr2=0x7f3f7c06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7c0ae8f0 0x7f3f7c06e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7f3f80ae0700 1 -- 172.21.15.115:0/374241297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0094f0 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7ff01daa6700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.327+0000 7f01f27fc700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6808/115950 conn(0x7f01e007f760 0x7f01e0081b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.327+0000 7f01f27fc700 10 monclient: get_auth_request con 0x7f01e007f760 auth_method 0 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.326+0000 7ff01daa6700 10 monclient: monmap: 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.327+0000 7ff01daa6700 1 librados: starting msgr at 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.327+0000 7ff01daa6700 1 librados: starting objecter 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.327+0000 7f01f27fc700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6808/115950 conn(0x7f01e007f760 0x7f01e0081b70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7f01daffd700 10 client.4235.objecter ms_handle_connect 0x7f01e007f760 2023-09-30T01:03:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7f3f80ae0700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f7c0aecc0 0x7f3f7c14d9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009fa0 tx=0x7f3f70009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7f3f63fff700 10 client.?.objecter ms_handle_connect 0x7f3f7c0aecc0 2023-09-30T01:03:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7f3f63fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7ff01daa6700 1 Processor -- start 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.328+0000 7f0985ffb700 4 set_mon_vals no callback set 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 10 client.4305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 15 client.4305.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 20 client.4305.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.329+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f098806eba0 con 0x7f098813d7c0 2023-09-30T01:03:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 -- start start 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 librados: setting wanted keys 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 librados: calling monclient init 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient: init 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 10 client.4291.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 20 client.4291.objecter _get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 20 client.4291.objecter _get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 20 client.4291.objecter get_session s=0x7f797c002880 osd=5 3 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 15 client.4291.objecter _session_command_op_assign 5 2 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 4 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 10 client.4291.objecter _send_command 2 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 --> v2:172.21.15.157:6804/115847 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f797c001180 con 0x7f797c002f70 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== osd.1 v2:172.21.15.115:6808/115950 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f01e00821e0 con 0x7f01e007f760 2023-09-30T01:03:36.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f01daffd700 10 client.4235.objecter ms_dispatch 0x7f01f4001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f01daffd700 10 client.4235.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f01daffd700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 4 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f01daffd700 15 client.4235.objecter _session_command_op_remove 1 1 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70020520 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f7991ffb700 1 -- 172.21.15.115:0/960600190 <== osd.5 v2:172.21.15.157:6804/115847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f797c001180 con 0x7f797c002f70 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f7991ffb700 10 client.4291.objecter ms_dispatch 0x7f799c001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f7991ffb700 10 client.4291.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f7991ffb700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 4 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f7991ffb700 15 client.4291.objecter _session_command_op_remove 5 2 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: dump: 2023-09-30T01:03:36.485 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70020680 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70020a40 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f63fff700 10 monclient: dump: 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.487 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f7a7fc700 4 set_mon_vals no callback set 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 5 monclient: authenticate success, global_id 4308 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 10 monclient: _renew_subs 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c06e9a0 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 10 monclient: _renew_subs 2023-09-30T01:03:36.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c14dee0 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7f3f82543700 1 librados: init done 2023-09-30T01:03:36.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff018085ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-b con 0x7ff000004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01807a4e0 0x7ff0181412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-c con 0x7ff01807a4e0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 0x7ff0181417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): picked mon.noname-a con 0x7ff01807cf60 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01807b1e0 con 0x7ff01807cf60 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018086a30 con 0x7ff000004810 2023-09-30T01:03:36.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01807b700 con 0x7ff01807a4e0 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.330+0000 7ff01daa6700 10 monclient(hunting): authenticate will time out at 1077.780707s 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 0x7ff0181417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 0x7ff0181417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44126/0 (socket says 172.21.15.115:44126) 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 -- 172.21.15.115:0/76681891 learned_addr learned my addr 172.21.15.115:0/76681891 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): get_auth_request con 0x7ff01807cf60 auth_method 0 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 10 librados: watch_flush enter 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 10 librados: watch_flush exit 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 10 client.4291.objecter close_session for osd.5 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 >> v2:172.21.15.157:6804/115847 conn(0x7f797c002f70 msgr2=0x7f797c005420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.157:6804/115847 conn(0x7f797c002f70 0x7f797c005420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 20 client.4291.objecter put_session s=0x7f797c002880 osd=5 1 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 20 client.4291.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 10 client.4291.objecter successfully canceled tick 2023-09-30T01:03:36.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 -- 172.21.15.115:0/76681891 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01807a4e0 msgr2=0x7ff0181412a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 --2- 172.21.15.115:0/76681891 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff01807a4e0 0x7ff0181412a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 -- 172.21.15.115:0/76681891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 msgr2=0x7ff018085ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 --2- 172.21.15.115:0/76681891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff000004810 0x7ff018085ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7ff017fff700 1 -- 172.21.15.115:0/76681891 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01807b8c0 con 0x7ff01807cf60 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.331+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 >> v2:172.21.15.115:6816/115635 conn(0x7f7980068950 msgr2=0x7f798006ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/960600190 >> v2:172.21.15.115:6816/115635 conn(0x7f7980068950 0x7f798006ae00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f798c0089b0 tx=0x7f798c01b000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 10 monclient: shutdown 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 msgr2=0x7f799c13afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 1 --2- 172.21.15.115:0/960600190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799c13db30 0x7f799c13afc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f799800b520 tx=0x7f79980090f0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a223d700 1 -- 172.21.15.115:0/960600190 reap_dead start 2023-09-30T01:03:36.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 shutdown_connections 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 >> 172.21.15.115:0/960600190 conn(0x7f799c067bd0 msgr2=0x7f799c0af440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7ff017fff700 1 --2- 172.21.15.115:0/76681891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 0x7ff0181417e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff01806b550 tx=0x7ff008006e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 client.?.objecter ms_handle_connect 0x7ff01807cf60 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 10 client.4308.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 15 client.4308.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 20 client.4308.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f7c06eb30 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f0980007300 con 0x7f098813d7c0 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080060d0 con 0x7ff01807cf60 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: dump: 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.498 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff008006230 con 0x7ff01807cf60 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080065d0 con 0x7ff01807cf60 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7fefff7fe700 10 monclient: dump: 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.500 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6816/115635 conn(0x7f0968068800 0x7f096806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7ff01daa6700 5 monclient: authenticate success, global_id 4311 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.332+0000 7ff015ffb700 4 set_mon_vals no callback set 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 10 monclient: _renew_subs 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff018144830 con 0x7ff01807cf60 2023-09-30T01:03:36.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 10 monclient: _renew_subs 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff018144cd0 con 0x7ff01807cf60 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7ff01daa6700 1 librados: init done 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0980084bf0 con 0x7f098813d7c0 2023-09-30T01:03:36.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 10 client.4305.objecter ms_dispatch 0x7f0988001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 3 client.4305.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 3 client.4305.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 10 client.4305.objecter checking command 1 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6812/115952 conn(0x7f096806e770 0x7f0968070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter _get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 15 client.4305.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter _get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 15 client.4305.objecter _session_command_op_assign 3 1 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 4 2023-09-30T01:03:36.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 10 client.4305.objecter _send_command 1 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 --> v2:172.21.15.115:6812/115952 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09680711f0 con 0x7f096806e770 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 20 client.4305.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0980084e90 con 0x7f098813d7c0 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f097f7fe700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f0986ffd700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6812/115952 conn(0x7f096806e770 0x7f0968070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f0986ffd700 10 monclient: get_auth_request con 0x7f096806e770 auth_method 0 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.333+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f3f70022410 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6816/115635 conn(0x7f3f68068800 0x7f3f6806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3f70094d10 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 10 client.4308.objecter ms_dispatch 0x7f3f7c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 3 client.4308.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 3 client.4308.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 10 client.4308.objecter checking command 1 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6804/115951 conn(0x7f3f6806e770 0x7f3f68070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter _get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 15 client.4308.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter _get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 15 client.4308.objecter _session_command_op_assign 2 1 2023-09-30T01:03:36.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 4 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 10 client.4308.objecter _send_command 1 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 --> v2:172.21.15.115:6804/115951 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f680711f0 con 0x7f3f6806e770 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f63fff700 20 client.4308.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f7bfff700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6816/115635 conn(0x7f3f68068800 0x7f3f6806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7f3f7bfff700 10 monclient: get_auth_request con 0x7f3f68068800 auth_method 0 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 15 client.4311.objecter _session_command_op_assign -1 1 2023-09-30T01:03:36.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 20 client.4311.objecter _get_session osd=-1 returning homeless 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.334+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff01807b350 con 0x7ff01807cf60 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 shutdown_connections 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f79a3ca0700 1 -- 172.21.15.115:0/960600190 wait complete. 2023-09-30T01:03:36.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f3f63fff700 10 client.4308.objecter ms_handle_connect 0x7f3f68068800 2023-09-30T01:03:36.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f3f7bfff700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6816/115635 conn(0x7f3f68068800 0x7f3f6806acb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00a010 tx=0x7f3f6c01b000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f79a3ca0700 1 librados: shutdown 2023-09-30T01:03:36.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f70093980 con 0x7f3f7c0aecc0 2023-09-30T01:03:36.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f3f63fff700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f3f7b7fe700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6804/115951 conn(0x7f3f6806e770 0x7f3f68070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f3f7b7fe700 10 monclient: get_auth_request con 0x7f3f6806e770 auth_method 0 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f3f63fff700 10 client.4308.objecter ms_handle_connect 0x7f3f6806e770 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f3f7b7fe700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6804/115951 conn(0x7f3f6806e770 0x7f3f68070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 10 client.4306.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 15 client.4306.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 20 client.4306.objecter _get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 20 client.4306.objecter _get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 15 client.4306.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 20 client.4306.objecter get_session s=0x7f56a006e140 osd=0 3 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 15 client.4306.objecter _session_command_op_assign 0 2 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 4 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 10 client.4306.objecter _send_command 2 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 --> v2:172.21.15.115:6800/115935 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f56b413cfa0 con 0x7f56a006e7f0 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f569affd700 1 -- 172.21.15.115:0/3584668813 <== osd.0 v2:172.21.15.115:6800/115935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f56b413cfa0 con 0x7f56a006e7f0 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f569affd700 10 client.4306.objecter ms_dispatch 0x7f56b4001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f569affd700 10 client.4306.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f569affd700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 4 2023-09-30T01:03:36.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f569affd700 15 client.4306.objecter _session_command_op_remove 0 2 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== osd.2 v2:172.21.15.115:6804/115951 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3f680711f0 con 0x7f3f6806e770 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f3f63fff700 10 client.4308.objecter ms_dispatch 0x7f3f7c001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f3f63fff700 10 client.4308.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f3f63fff700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 4 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f3f63fff700 15 client.4308.objecter _session_command_op_remove 2 1 2023-09-30T01:03:36.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7ff008003a20 con 0x7ff01807cf60 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 10 client.4299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 15 client.4299.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 20 client.4299.objecter _get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 20 client.4299.objecter _get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 15 client.4299.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 20 client.4299.objecter get_session s=0x7f1de006e140 osd=6 3 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 15 client.4299.objecter _session_command_op_assign 6 2 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 4 2023-09-30T01:03:36.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 10 client.4299.objecter _send_command 2 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 --> v2:172.21.15.157:6808/115848 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1e000a5510 con 0x7f1de006e7f0 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 10 client.4294.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 15 client.4294.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 20 client.4294.objecter _get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 20 client.4294.objecter _get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 15 client.4294.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 20 client.4294.objecter get_session s=0x7f60d4011e90 osd=7 3 2023-09-30T01:03:36.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 15 client.4294.objecter _session_command_op_assign 7 2 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 4 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 10 client.4294.objecter _send_command 2 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.337+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 --> v2:172.21.15.157:6812/115891 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60c40011c0 con 0x7f60d006e750 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 10 librados: watch_flush enter 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 10 librados: watch_flush exit 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 10 client.4306.objecter close_session for osd.0 2023-09-30T01:03:36.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6800/115935 conn(0x7f56a006e7f0 msgr2=0x7f56a0070c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6800/115935 conn(0x7f56a006e7f0 0x7f56a0070c00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 20 client.4306.objecter put_session s=0x7f56a006e140 osd=0 1 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 20 client.4306.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 10 client.4306.objecter successfully canceled tick 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.115:6816/115635 conn(0x7ff004068800 0x7ff00406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff008082df0 con 0x7ff01807cf60 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 10 client.4311.objecter ms_dispatch 0x7ff018001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 3 client.4311.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 3 client.4311.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 10 client.4311.objecter checking command 1 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.157:6800/115846 conn(0x7ff00406e770 0x7ff004070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter _get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 15 client.4311.objecter _session_command_op_remove -1 1 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter _get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 15 client.4311.objecter _session_command_op_assign 4 1 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 4 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 10 client.4311.objecter _send_command 1 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 --> v2:172.21.15.157:6800/115846 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0040711f0 con 0x7ff00406e770 2023-09-30T01:03:36.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 20 client.4311.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff008085300 con 0x7ff01807cf60 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7fefff7fe700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6816/115635 conn(0x7f56a0068880 msgr2=0x7f56a006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.338+0000 7f56b992c700 1 --2- 172.21.15.115:0/3584668813 >> v2:172.21.15.115:6816/115635 conn(0x7f56a0068880 0x7f56a006ad30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00acd0 tx=0x7f56ac009350 comp rx=0 tx=0).stop 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 10 monclient: shutdown 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 msgr2=0x7f56b40a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 --2- 172.21.15.115:0/3584668813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56b40a64c0 0x7f56b40a6910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56a4001620 tx=0x7f56a4001d90 comp rx=0 tx=0).stop 2023-09-30T01:03:36.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b37fe700 1 -- 172.21.15.115:0/3584668813 reap_dead start 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 shutdown_connections 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 >> 172.21.15.115:0/3584668813 conn(0x7f56b4068900 msgr2=0x7f56b40abf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f0986ffd700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6812/115952 conn(0x7f096806e770 0x7f0968070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f097f7fe700 10 client.4305.objecter ms_handle_connect 0x7f096806e770 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f09877fe700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6816/115635 conn(0x7f0968068800 0x7f096806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.335+0000 7f09877fe700 10 monclient: get_auth_request con 0x7f0968068800 auth_method 0 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f097f7fe700 10 client.4305.objecter ms_handle_connect 0x7f0968068800 2023-09-30T01:03:36.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.336+0000 7f09877fe700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6816/115635 conn(0x7f0968068800 0x7f096806acb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09880b4890 tx=0x7f0970017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 shutdown_connections 2023-09-30T01:03:36.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 -- 172.21.15.115:0/3584668813 wait complete. 2023-09-30T01:03:36.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7ff016ffd700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.157:6800/115846 conn(0x7ff00406e770 0x7ff004070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7ff016ffd700 10 monclient: get_auth_request con 0x7ff00406e770 auth_method 0 2023-09-30T01:03:36.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f56b992c700 1 librados: shutdown 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f60caffd700 1 -- 172.21.15.115:0/3021478179 <== osd.7 v2:172.21.15.157:6812/115891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f60c40011c0 con 0x7f60d006e750 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f60caffd700 10 client.4294.objecter ms_dispatch 0x7f60e4001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f60caffd700 10 client.4294.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f60caffd700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 4 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f60caffd700 15 client.4294.objecter _session_command_op_remove 7 2 2023-09-30T01:03:36.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7ff016ffd700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.157:6800/115846 conn(0x7ff00406e770 0x7ff004070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7fefff7fe700 10 client.4311.objecter ms_handle_connect 0x7ff00406e770 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f1df67fc700 1 -- 172.21.15.115:0/720649932 <== osd.6 v2:172.21.15.157:6808/115848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1df8001e30 con 0x7f1de006e7f0 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f1df67fc700 10 client.4299.objecter ms_dispatch 0x7f1e00001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f1df67fc700 10 client.4299.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f1df67fc700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 4 2023-09-30T01:03:36.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7f1df67fc700 15 client.4299.objecter _session_command_op_remove 6 2 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7ff0177fe700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.115:6816/115635 conn(0x7ff004068800 0x7ff00406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.339+0000 7ff0177fe700 10 monclient: get_auth_request con 0x7ff004068800 auth_method 0 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== osd.3 v2:172.21.15.115:6812/115952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f09680711f0 con 0x7f096806e770 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f097f7fe700 10 client.4305.objecter ms_dispatch 0x7f0988001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f097f7fe700 10 client.4305.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f097f7fe700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 4 2023-09-30T01:03:36.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f097f7fe700 15 client.4305.objecter _session_command_op_remove 3 1 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 10 librados: watch_flush enter 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 10 librados: watch_flush exit 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 10 client.4299.objecter close_session for osd.6 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 >> v2:172.21.15.157:6808/115848 conn(0x7f1de006e7f0 msgr2=0x7f1de0070c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.157:6808/115848 conn(0x7f1de006e7f0 0x7f1de0070c00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 20 client.4299.objecter put_session s=0x7f1de006e140 osd=6 1 2023-09-30T01:03:36.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 20 client.4299.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7f1e04bc1700 10 client.4299.objecter successfully canceled tick 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== osd.4 v2:172.21.15.157:6800/115846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff0040711f0 con 0x7ff00406e770 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 10 client.4311.objecter ms_dispatch 0x7ff018001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 10 client.4311.objecter _finish_command 1 = system:0 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 4 2023-09-30T01:03:36.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 15 client.4311.objecter _session_command_op_remove 4 1 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7ff0177fe700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.115:6816/115635 conn(0x7ff004068800 0x7ff00406acb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff01000acf0 tx=0x7ff010009370 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.340+0000 7fefff7fe700 10 client.4311.objecter ms_handle_connect 0x7ff004068800 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 >> v2:172.21.15.115:6816/115635 conn(0x7f1de0068880 msgr2=0x7f1de006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/720649932 >> v2:172.21.15.115:6816/115635 conn(0x7f1de0068880 0x7f1de006ad30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1e000b08d0 tx=0x7f1df0016000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 10 monclient: shutdown 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 msgr2=0x7f1e000a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 --2- 172.21.15.115:0/720649932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1e0006bc10 0x7f1e000a5fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1de800b840 tx=0x7f1de80091d0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1dfed9d700 1 -- 172.21.15.115:0/720649932 reap_dead start 2023-09-30T01:03:36.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 shutdown_connections 2023-09-30T01:03:36.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 >> 172.21.15.115:0/720649932 conn(0x7f1e000699e0 msgr2=0x7f1e0013f620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.341+0000 7f60e9987700 10 librados: watch_flush enter 2023-09-30T01:03:36.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 10 librados: watch_flush exit 2023-09-30T01:03:36.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 10 client.4294.objecter close_session for osd.7 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 >> v2:172.21.15.157:6812/115891 conn(0x7f60d006e750 msgr2=0x7f60d0070b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.157:6812/115891 conn(0x7f60d006e750 0x7f60d0070b60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 20 client.4294.objecter put_session s=0x7f60d4011e90 osd=7 1 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 20 client.4294.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 10 client.4294.objecter successfully canceled tick 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 >> v2:172.21.15.115:6816/115635 conn(0x7f60d0068950 msgr2=0x7f60d006ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 --2- 172.21.15.115:0/3021478179 >> v2:172.21.15.115:6816/115635 conn(0x7f60d0068950 0x7f60d006ae00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f60e4145210 tx=0x7f60d8008040 comp rx=0 tx=0).stop 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 10 monclient: shutdown 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145e00 msgr2=0x7f60e41426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e9987700 1 --2- 172.21.15.115:0/3021478179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60e4145e00 0x7f60e41426f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60d4013000 tx=0x7f60d4017000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.347+0000 7f60e37fe700 1 -- 172.21.15.115:0/3021478179 reap_dead start 2023-09-30T01:03:36.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.351+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 shutdown_connections 2023-09-30T01:03:36.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.351+0000 7f1e04bc1700 1 -- 172.21.15.115:0/720649932 wait complete. 2023-09-30T01:03:36.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.351+0000 7f1e04bc1700 1 librados: shutdown 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 shutdown_connections 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 >> 172.21.15.115:0/3021478179 conn(0x7f60e40b0540 msgr2=0x7f60e40ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 shutdown_connections 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 10 client.4235.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 15 client.4235.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 20 client.4235.objecter _get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f60e9987700 1 -- 172.21.15.115:0/3021478179 wait complete. 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 20 client.4235.objecter _get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 15 client.4235.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 20 client.4235.objecter get_session s=0x7f01ec008e90 osd=1 3 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 15 client.4235.objecter _session_command_op_assign 1 2 2023-09-30T01:03:36.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 4 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 10 client.4235.objecter _send_command 2 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 --> v2:172.21.15.115:6808/115950 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01f40aab40 con 0x7f01e007f760 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.352+0000 7f60e9987700 1 librados: shutdown 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01daffd700 1 -- 172.21.15.115:0/1188621722 <== osd.1 v2:172.21.15.115:6808/115950 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f01f40aab40 con 0x7f01e007f760 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01daffd700 10 client.4235.objecter ms_dispatch 0x7f01f4001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01daffd700 10 client.4235.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01daffd700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 4 2023-09-30T01:03:36.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01daffd700 15 client.4235.objecter _session_command_op_remove 1 2 2023-09-30T01:03:36.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 10 librados: watch_flush enter 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 10 librados: watch_flush exit 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 10 client.4235.objecter close_session for osd.1 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6808/115950 conn(0x7f01e007f760 msgr2=0x7f01e0081b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6808/115950 conn(0x7f01e007f760 0x7f01e0081b70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 20 client.4235.objecter put_session s=0x7f01ec008e90 osd=1 1 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 20 client.4235.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.353+0000 7f01f97d1700 10 client.4235.objecter successfully canceled tick 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6816/115635 conn(0x7f01e0079960 msgr2=0x7f01e007be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 --2- 172.21.15.115:0/1188621722 >> v2:172.21.15.115:6816/115635 conn(0x7f01e0079960 0x7f01e007be10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01e40098b0 tx=0x7f01e401d000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 10 monclient: shutdown 2023-09-30T01:03:36.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 msgr2=0x7f01f4151310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 --2- 172.21.15.115:0/1188621722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f01f40b0e10 0x7f01f4151310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009fd0 tx=0x7f01ec00b1b0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f37fe700 1 -- 172.21.15.115:0/1188621722 reap_dead start 2023-09-30T01:03:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 shutdown_connections 2023-09-30T01:03:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.354+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 >> 172.21.15.115:0/1188621722 conn(0x7f01f4067bd0 msgr2=0x7f01f406b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.535 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:36.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 10 client.4308.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 15 client.4308.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 20 client.4308.objecter _get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 20 client.4308.objecter _get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 15 client.4308.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 20 client.4308.objecter get_session s=0x7f3f6806e0c0 osd=2 3 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 15 client.4308.objecter _session_command_op_assign 2 2 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 4 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 10 client.4308.objecter _send_command 2 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 --> v2:172.21.15.115:6804/115951 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3f7c072960 con 0x7f3f6806e770 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f63fff700 1 -- 172.21.15.115:0/374241297 <== osd.2 v2:172.21.15.115:6804/115951 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3f7c072960 con 0x7f3f6806e770 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f63fff700 10 client.4308.objecter ms_dispatch 0x7f3f7c001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f63fff700 10 client.4308.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f63fff700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 4 2023-09-30T01:03:36.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.356+0000 7f3f63fff700 15 client.4308.objecter _session_command_op_remove 2 2 2023-09-30T01:03:36.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 10 librados: watch_flush enter 2023-09-30T01:03:36.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 10 librados: watch_flush exit 2023-09-30T01:03:36.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 10 client.4308.objecter close_session for osd.2 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6804/115951 conn(0x7f3f6806e770 msgr2=0x7f3f68070b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6804/115951 conn(0x7f3f6806e770 0x7f3f68070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 20 client.4308.objecter put_session s=0x7f3f6806e0c0 osd=2 1 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 20 client.4308.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 10 client.4308.objecter successfully canceled tick 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6816/115635 conn(0x7f3f68068800 msgr2=0x7f3f6806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 --2- 172.21.15.115:0/374241297 >> v2:172.21.15.115:6816/115635 conn(0x7f3f68068800 0x7f3f6806acb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00a010 tx=0x7f3f6c01b000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 10 monclient: shutdown 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f7c0aecc0 msgr2=0x7f3f7c14d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f82543700 1 --2- 172.21.15.115:0/374241297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3f7c0aecc0 0x7f3f7c14d9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009fa0 tx=0x7f3f70009120 comp rx=0 tx=0).stop 2023-09-30T01:03:36.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.357+0000 7f3f80ae0700 1 -- 172.21.15.115:0/374241297 reap_dead start 2023-09-30T01:03:36.540 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:36.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:03:36.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.358+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 shutdown_connections 2023-09-30T01:03:36.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.358+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 >> 172.21.15.115:0/374241297 conn(0x7f3f7c06a490 msgr2=0x7f3f7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.358+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 shutdown_connections 2023-09-30T01:03:36.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.358+0000 7f3f82543700 1 -- 172.21.15.115:0/374241297 wait complete. 2023-09-30T01:03:36.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.358+0000 7f3f82543700 1 librados: shutdown 2023-09-30T01:03:36.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.360+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 shutdown_connections 2023-09-30T01:03:36.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.360+0000 7f01f97d1700 1 -- 172.21.15.115:0/1188621722 wait complete. 2023-09-30T01:03:36.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.360+0000 7f01f97d1700 1 librados: shutdown 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 10 client.4305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 15 client.4305.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 20 client.4305.objecter _get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 20 client.4305.objecter _get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 15 client.4305.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 20 client.4305.objecter get_session s=0x7f096806e0c0 osd=3 3 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 15 client.4305.objecter _session_command_op_assign 3 2 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 4 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 10 client.4305.objecter _send_command 2 2023-09-30T01:03:36.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.369+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 --> v2:172.21.15.115:6812/115952 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0988072960 con 0x7f096806e770 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.372+0000 7f097f7fe700 1 -- 172.21.15.115:0/3294498983 <== osd.3 v2:172.21.15.115:6812/115952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0988072960 con 0x7f096806e770 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.372+0000 7f097f7fe700 10 client.4305.objecter ms_dispatch 0x7f0988001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.372+0000 7f097f7fe700 10 client.4305.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.372+0000 7f097f7fe700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 4 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.372+0000 7f097f7fe700 15 client.4305.objecter _session_command_op_remove 3 2 2023-09-30T01:03:36.544 INFO:teuthology.orchestra.run.smithi115.stdout:55834574852 2023-09-30T01:03:36.545 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T01:03:36.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 15 client.4311.objecter _session_command_op_assign -1 2 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 20 client.4311.objecter _get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 20 client.4311.objecter _get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 15 client.4311.objecter _session_command_op_remove -1 2 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 20 client.4311.objecter get_session s=0x7ff00406e0c0 osd=4 3 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 15 client.4311.objecter _session_command_op_assign 4 2 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 4 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 10 client.4311.objecter _send_command 2 2023-09-30T01:03:36.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 --> v2:172.21.15.157:6800/115846 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff018072960 con 0x7ff00406e770 2023-09-30T01:03:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 10 librados: watch_flush enter 2023-09-30T01:03:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 10 librados: watch_flush exit 2023-09-30T01:03:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 10 client.4305.objecter close_session for osd.3 2023-09-30T01:03:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7fefff7fe700 1 -- 172.21.15.115:0/76681891 <== osd.4 v2:172.21.15.157:6800/115846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff018072960 con 0x7ff00406e770 2023-09-30T01:03:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7fefff7fe700 10 client.4311.objecter ms_dispatch 0x7ff018001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7fefff7fe700 10 client.4311.objecter _finish_command 2 = system:0 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7fefff7fe700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 4 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7fefff7fe700 15 client.4311.objecter _session_command_op_remove 4 2 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6812/115952 conn(0x7f096806e770 msgr2=0x7f0968070b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6812/115952 conn(0x7f096806e770 0x7f0968070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 20 client.4305.objecter put_session s=0x7f096806e0c0 osd=3 1 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 20 client.4305.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 10 client.4305.objecter successfully canceled tick 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6816/115635 conn(0x7f0968068800 msgr2=0x7f096806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 --2- 172.21.15.115:0/3294498983 >> v2:172.21.15.115:6816/115635 conn(0x7f0968068800 0x7f096806acb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09880b4890 tx=0x7f0970017000 comp rx=0 tx=0).stop 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 10 monclient: shutdown 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 msgr2=0x7f098813e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.373+0000 7f098e02f700 1 --2- 172.21.15.115:0/3294498983 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f098813d7c0 0x7f098813e740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f098000cbc0 tx=0x7f09800019c0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7f0987fff700 1 -- 172.21.15.115:0/3294498983 reap_dead start 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 10 librados: watch_flush enter 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 10 librados: watch_flush exit 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 10 client.4311.objecter close_session for osd.4 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 >> v2:172.21.15.157:6800/115846 conn(0x7ff00406e770 msgr2=0x7ff004070b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.157:6800/115846 conn(0x7ff00406e770 0x7ff004070b80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 20 client.4311.objecter put_session s=0x7ff00406e0c0 osd=4 1 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 20 client.4311.objecter shutdown clearing up homeless session... 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 10 client.4311.objecter successfully canceled tick 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 >> v2:172.21.15.115:6816/115635 conn(0x7ff004068800 msgr2=0x7ff00406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 --2- 172.21.15.115:0/76681891 >> v2:172.21.15.115:6816/115635 conn(0x7ff004068800 0x7ff00406acb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff01000acf0 tx=0x7ff010009370 comp rx=0 tx=0).stop 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 10 monclient: shutdown 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 msgr2=0x7ff0181417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 --2- 172.21.15.115:0/76681891 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff01807cf60 0x7ff0181417e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff01806b550 tx=0x7ff008006e70 comp rx=0 tx=0).stop 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff017fff700 1 -- 172.21.15.115:0/76681891 reap_dead start 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 shutdown_connections 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 >> 172.21.15.115:0/76681891 conn(0x7ff0180699e0 msgr2=0x7ff0180aba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 shutdown_connections 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 -- 172.21.15.115:0/76681891 wait complete. 2023-09-30T01:03:36.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.374+0000 7ff01daa6700 1 librados: shutdown 2023-09-30T01:03:36.553 INFO:teuthology.orchestra.run.smithi115.stdout:55834574852 2023-09-30T01:03:36.553 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T01:03:36.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.377+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 shutdown_connections 2023-09-30T01:03:36.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.377+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 >> 172.21.15.115:0/3294498983 conn(0x7f098806a460 msgr2=0x7f098813fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.377+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 shutdown_connections 2023-09-30T01:03:36.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.377+0000 7f098e02f700 1 -- 172.21.15.115:0/3294498983 wait complete. 2023-09-30T01:03:36.555 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:36.555 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:03:36.556 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:36.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:03:36.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.380+0000 7f098e02f700 1 librados: shutdown 2023-09-30T01:03:36.557 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:36.557 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:03:36.558 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:36.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:03:36.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.775+0000 7fdf52b70700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.775+0000 7fdf52b70700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.775+0000 7fdf52b70700 10 monclient: monmap: 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.776116+0000 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.776116+0000 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.778 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.779+0000 7fdf52b70700 1 Processor -- start 2023-09-30T01:03:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 -- start start 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient: init 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 0x7fdf4c06ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-c con 0x7fdf4c06e400 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf4c13d6d0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-b con 0x7fdf4c13db30 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c06bb30 con 0x7fdf4c13d6d0 2023-09-30T01:03:36.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c06d840 con 0x7fdf4c06e400 2023-09-30T01:03:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0aab40 con 0x7fdf4c13db30 2023-09-30T01:03:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.780+0000 7fdf52b70700 10 monclient(hunting): authenticate will time out at 1078.230997s 2023-09-30T01:03:36.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59056/0 (socket says 172.21.15.115:59056) 2023-09-30T01:03:36.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 -- 172.21.15.115:0/3655062817 learned_addr learned my addr 172.21.15.115:0/3655062817 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): get_auth_request con 0x7fdf4c13db30 auth_method 0 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5090c700 1 --2- 172.21.15.115:0/3655062817 >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 0x7fdf4c06ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5090c700 10 monclient(hunting): get_auth_request con 0x7fdf4c06e400 auth_method 0 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5090c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5090c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5090c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 -- 172.21.15.115:0/3655062817 >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 msgr2=0x7fdf4c06ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 --2- 172.21.15.115:0/3655062817 >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 0x7fdf4c06ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.781+0000 7fdf5110d700 1 -- 172.21.15.115:0/3655062817 >> v1:172.21.15.115:6789/0 conn(0x7fdf4c13d6d0 legacy=0x7fdf4c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 1 -- 172.21.15.115:0/3655062817 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c06da40 con 0x7fdf4c13db30 2023-09-30T01:03:36.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5090c700 1 --2- 172.21.15.115:0/3655062817 >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 0x7fdf4c06ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf5110d700 1 --2- 172.21.15.115:0/3655062817 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c007190 tx=0x7fdf3c009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e88502ff46ff1b5a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 1 -- 172.21.15.115:0/3655062817 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0092e0 con 0x7fdf4c13db30 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: dump: 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.786 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 1 -- 172.21.15.115:0/3655062817 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c009440 con 0x7fdf4c13db30 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 1 -- 172.21.15.115:0/3655062817 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c007660 con 0x7fdf4c13db30 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf4b7fe700 10 monclient: dump: 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf52b70700 5 monclient: authenticate success, global_id 4250 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf52b70700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.782+0000 7fdf52b70700 4 set_mon_vals no callback set 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 10 monclient: shutdown 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 -- 172.21.15.115:0/3655062817 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 msgr2=0x7fdf4c06c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 --2- 172.21.15.115:0/3655062817 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c007190 tx=0x7fdf3c009b70 comp rx=0 tx=0).stop 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 -- 172.21.15.115:0/3655062817 shutdown_connections 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 --2- 172.21.15.115:0/3655062817 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13db30 0x7fdf4c06c4d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 --2- 172.21.15.115:0/3655062817 >> v2:172.21.15.157:3300/0 conn(0x7fdf4c06e400 0x7fdf4c06ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 -- 172.21.15.115:0/3655062817 >> 172.21.15.115:0/3655062817 conn(0x7fdf4c067bd0 msgr2=0x7fdf4c0b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 -- 172.21.15.115:0/3655062817 shutdown_connections 2023-09-30T01:03:36.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.783+0000 7fdf52b70700 1 -- 172.21.15.115:0/3655062817 wait complete. 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 10 monclient: monmap: 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.791 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 librados: starting msgr at 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 librados: starting objecter 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 Processor -- start 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 -- start start 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 librados: setting wanted keys 2023-09-30T01:03:36.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 1 librados: calling monclient init 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.784+0000 7fdf52b70700 10 monclient: init 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 0x7fdf4c14c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-a con 0x7fdf4c06e400 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 0x7fdf4c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-c con 0x7fdf4c13d6d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf4c13db30 0x7fdf4c0a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): picked mon.noname-b con 0x7fdf4c13db30 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0a7c40 con 0x7fdf4c06e400 2023-09-30T01:03:36.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0a5de0 con 0x7fdf4c13db30 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0a6d80 con 0x7fdf4c13d6d0 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7fdf52b70700 10 monclient(hunting): authenticate will time out at 1078.235824s 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 0x7fdf4c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 0x7fdf4c0a6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59068/0 (socket says 172.21.15.115:59068) 2023-09-30T01:03:36.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 1 -- 172.21.15.115:0/1989251803 learned_addr learned my addr 172.21.15.115:0/1989251803 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): get_auth_request con 0x7fdf4c13d6d0 auth_method 0 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 0x7fdf4c14c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): get_auth_request con 0x7fdf4c06e400 auth_method 0 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf4c13db30 0x7fdf4c0a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): get_auth_request con 0x7fdf4c13db30 auth_method 0 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5110d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf4bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 -- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 msgr2=0x7fdf4c0a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 0x7fdf4c0a6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 -- 172.21.15.115:0/1989251803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf4c13db30 msgr2=0x7fdf4c0a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf4c13db30 0x7fdf4c0a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.786+0000 7fdf5090c700 1 -- 172.21.15.115:0/1989251803 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c00d040 con 0x7fdf4c06e400 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7f1bf91ed700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7f1bf91ed700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.785+0000 7f1bf91ed700 10 monclient: monmap: 2023-09-30T01:03:36.798 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.786854+0000 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.786854+0000 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.799 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf5090c700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 0x7fdf4c14c8a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000dd70 tx=0x7fdf40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 client.?.objecter ms_handle_connect 0x7fdf4c06e400 2023-09-30T01:03:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7f1bf91ed700 1 Processor -- start 2023-09-30T01:03:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4000a8a0 con 0x7fdf4c06e400 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: dump: 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.801 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4000aa00 con 0x7fdf4c06e400 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4000adc0 con 0x7fdf4c06e400 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7f1bf91ed700 1 -- start start 2023-09-30T01:03:36.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7f1bf91ed700 10 monclient: init 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf48ff9700 10 monclient: dump: 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.803 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 5 monclient: authenticate success, global_id 4314 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 10 monclient: _renew_subs 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c14d2f0 con 0x7fdf4c06e400 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 10 monclient: _renew_subs 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c0a5a80 con 0x7fdf4c06e400 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fdf52b70700 1 librados: init done 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bf40b0c50 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-b con 0x7f1bf40b1020 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 0x7f1bf413e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413b900 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf406f560 con 0x7f1bf40b0c50 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf4142210 con 0x7f1bf413b900 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4142410 con 0x7f1bf40b1020 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.788+0000 7f1bf91ed700 10 monclient(hunting): authenticate will time out at 1078.238813s 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59078/0 (socket says 172.21.15.115:59078) 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 -- 172.21.15.115:0/3797279505 learned_addr learned my addr 172.21.15.115:0/3797279505 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): get_auth_request con 0x7f1bf40b1020 auth_method 0 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fecc1977700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fecc1977700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.787+0000 7fecc1977700 10 monclient: monmap: 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.807 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.788875+0000 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.788875+0000 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.808 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7fdf4affd700 4 set_mon_vals no callback set 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fdf4003b690 con 0x7fdf4c06e400 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 1 --2- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 0x7fdf3806ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdf40036070 con 0x7fdf4c06e400 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 10 client.4314.objecter ms_dispatch 0x7fdf4c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 3 client.4314.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 3 client.4314.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf52b70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf52b70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf52b70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf48ff9700 20 client.4314.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c0050b0 con 0x7fdf4c06e400 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7fecc1977700 1 Processor -- start 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7fecc1977700 1 -- start start 2023-09-30T01:03:36.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7fecc1977700 10 monclient: init 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 1 --2- 172.21.15.115:0/3797279505 >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 0x7f1bf413e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): get_auth_request con 0x7f1bf413b900 auth_method 0 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 -- 172.21.15.115:0/3797279505 >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 msgr2=0x7f1bf413e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 --2- 172.21.15.115:0/3797279505 >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 0x7f1bf413e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 -- 172.21.15.115:0/3797279505 >> v1:172.21.15.157:6789/0 conn(0x7f1bf40b0c50 legacy=0x7f1bf4141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 -- 172.21.15.115:0/3797279505 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf406b1f0 con 0x7f1bf40b1020 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf359e700 1 --2- 172.21.15.115:0/3797279505 >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 0x7f1bf413e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.789+0000 7f1bf259c700 1 --2- 172.21.15.115:0/3797279505 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1be40099b0 tx=0x7f1be4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3507ac63c16fba9a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 1 -- 172.21.15.115:0/3797279505 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4009ea0 con 0x7f1bf40b1020 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: dump: 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.813 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 1 -- 172.21.15.115:0/3797279505 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be400b860 con 0x7f1bf40b1020 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 1 -- 172.21.15.115:0/3797279505 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be400bc00 con 0x7f1bf40b1020 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf1d9b700 10 monclient: dump: 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.815 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 5 monclient: authenticate success, global_id 4256 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 4 set_mon_vals no callback set 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 10 monclient: shutdown 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3797279505 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 msgr2=0x7f1bf413b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.790+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3797279505 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1be40099b0 tx=0x7f1be4009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3797279505 shutdown_connections 2023-09-30T01:03:36.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3797279505 >> v2:172.21.15.157:3300/0 conn(0x7f1bf413b900 0x7f1bf413e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3797279505 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf40b1020 0x7f1bf413b3c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3797279505 >> 172.21.15.115:0/3797279505 conn(0x7f1bf4067bd0 msgr2=0x7f1bf40b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f6ff7ea6700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f6ff7ea6700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7f6ff7ea6700 10 monclient: monmap: 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.817 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.792661+0000 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.792661+0000 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.818 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fecbc0b0e10 0x7fecbc142470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-c con 0x7fecbc0b0e10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecbc06b280 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-a con 0x7fecbc0aaeb0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecbc06bdb0 con 0x7fecbc06b280 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecbc06c950 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecbc06d8e0 con 0x7fecbc0b0e10 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecc1977700 10 monclient(hunting): authenticate will time out at 1078.242134s 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44142/0 (socket says 172.21.15.115:44142) 2023-09-30T01:03:36.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 -- 172.21.15.115:0/2628015471 learned_addr learned my addr 172.21.15.115:0/2628015471 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): get_auth_request con 0x7fecbc0aaeb0 auth_method 0 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7fdf4bfff700 1 --2- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 0x7fdf3806ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.791+0000 7fdf4bfff700 10 monclient: get_auth_request con 0x7fdf38068950 auth_method 0 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fdf4bfff700 1 --2- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 0x7fdf3806ae00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c06fa80 tx=0x7fdf44009840 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fdf48ff9700 10 client.4314.objecter ms_handle_connect 0x7fdf38068950 2023-09-30T01:03:36.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbaffd700 1 --2- 172.21.15.115:0/2628015471 >> v2:172.21.15.157:3300/0 conn(0x7fecbc0b0e10 0x7fecbc142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbaffd700 10 monclient(hunting): get_auth_request con 0x7fecbc0b0e10 auth_method 0 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 -- 172.21.15.115:0/2628015471 >> v2:172.21.15.157:3300/0 conn(0x7fecbc0b0e10 msgr2=0x7fecbc142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 --2- 172.21.15.115:0/2628015471 >> v2:172.21.15.157:3300/0 conn(0x7fecbc0b0e10 0x7fecbc142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 -- 172.21.15.115:0/2628015471 >> v1:172.21.15.157:6789/0 conn(0x7fecbc06b280 legacy=0x7fecbc06c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecb9ffb700 1 -- 172.21.15.115:0/2628015471 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114573357 0 0) 0x7fecbc06bdb0 con 0x7fecbc06b280 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7fecbb7fe700 1 -- 172.21.15.115:0/2628015471 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc0b11e0 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7fecb9ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7fecbb7fe700 1 --2- 172.21.15.115:0/2628015471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0ac020 tx=0x7fecac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49548c37766cf9e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3797279505 shutdown_connections 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7feaa3ffe700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7feaa3ffe700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.792+0000 7feaa3ffe700 10 monclient: monmap: 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.793655+0000 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.793655+0000 2023-09-30T01:03:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f42ba66e700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f42ba66e700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f42ba66e700 10 monclient: monmap: 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.794179+0000 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.794179+0000 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f42ba66e700 1 Processor -- start 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7feaa3ffe700 1 Processor -- start 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7feaa3ffe700 1 -- start start 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7feaa3ffe700 10 monclient: init 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea9c06e400 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea9c06ec40 0x7fea9c13a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-c con 0x7fea9c06ec40 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c13a8f0 0x7fea9c13ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-b con 0x7fea9c13a8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9c06bb30 con 0x7fea9c06e400 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9c13b270 con 0x7fea9c06ec40 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c06d840 con 0x7fea9c13a8f0 2023-09-30T01:03:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa3ffe700 10 monclient(hunting): authenticate will time out at 1078.245350s 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3797279505 wait complete. 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f1bf91ed700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f1bf91ed700 10 monclient: monmap: 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f1bf91ed700 1 librados: starting msgr at 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f1bf91ed700 1 librados: starting objecter 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f1bf91ed700 1 Processor -- start 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.793+0000 7f6ff7ea6700 1 Processor -- start 2023-09-30T01:03:36.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f6ff7ea6700 1 -- start start 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f6ff7ea6700 10 monclient: init 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea9c06ec40 0x7fea9c13a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea9c06ec40 0x7fea9c13a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45518/0 (socket says 172.21.15.115:45518) 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 1 -- 172.21.15.115:0/2136843862 learned_addr learned my addr 172.21.15.115:0/2136843862 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): get_auth_request con 0x7fea9c06ec40 auth_method 0 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.794+0000 7f42ba66e700 1 -- start start 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient: init 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42b406b6f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f42b406c6f0 0x7f42b413a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-a con 0x7f42b406c6f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b413a760 0x7f42b413ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-b con 0x7f42b413a760 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b413b0a0 con 0x7f42b406b6f0 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b406d360 con 0x7f42b406c6f0 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406d640 con 0x7f42b413a760 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f42ba66e700 10 monclient(hunting): authenticate will time out at 1078.245942s 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f42b406c6f0 0x7f42b413a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f42b406c6f0 0x7f42b413a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44144/0 (socket says 172.21.15.115:44144) 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 -- 172.21.15.115:0/1939770717 learned_addr learned my addr 172.21.15.115:0/1939770717 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): get_auth_request con 0x7f42b406c6f0 auth_method 0 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7feaa1599700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2023-09-30T01:03:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 -- 172.21.15.115:0/2136843862 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c13a8f0 msgr2=0x7fea9c13ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 --2- 172.21.15.115:0/2136843862 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c13a8f0 0x7fea9c13ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 -- 172.21.15.115:0/2136843862 >> v1:172.21.15.115:6789/0 conn(0x7fea9c06e400 legacy=0x7fea9c13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 monclient: found mon.noname-c 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 -- 172.21.15.115:0/2136843862 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c13dde0 con 0x7fea9c06ec40 2023-09-30T01:03:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 -- start start 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 librados: setting wanted keys 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 librados: calling monclient init 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient: init 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bf40b0c50 0x7f1bf4152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40b0c50 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-b con 0x7f1bf40b1020 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf413b900 0x7f1bf40a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413b900 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40a7200 con 0x7f1bf40b0c50 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40a62f0 con 0x7f1bf40b1020 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40a8130 con 0x7f1bf413b900 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.795+0000 7f1bf91ed700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf91ed700 10 monclient(hunting): authenticate will time out at 1078.246008s 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7feaa1599700 1 --2- 172.21.15.115:0/2136843862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c06ec40 0x7fea9c13a3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009320 tx=0x7fea8c009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c2a6acfb900223 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42abfff700 1 -- 172.21.15.115:0/1939770717 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638072023 0 0) 0x7f42b413b0a0 con 0x7f42b406b6f0 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42abfff700 10 monclient(hunting): my global_id is 4342 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42abfff700 1 -- 172.21.15.115:0/1939770717 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a0003570 con 0x7f42b406b6f0 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2023-09-30T01:03:36.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 -- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b413a760 msgr2=0x7f42b413ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b413a760 0x7f42b413ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 -- 172.21.15.115:0/1939770717 >> v1:172.21.15.157:6789/0 conn(0x7f42b406b6f0 legacy=0x7f42b406e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f42b8e6b700 1 -- 172.21.15.115:0/1939770717 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b406d920 con 0x7f42b406c6f0 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45526/0 (socket says 172.21.15.115:45526) 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 1 -- 172.21.15.115:0/3807182329 learned_addr learned my addr 172.21.15.115:0/3807182329 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): get_auth_request con 0x7f1bf40b1020 auth_method 0 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bf40b0c50 0x7f1bf4152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): get_auth_request con 0x7f1bf40b0c50 auth_method 0 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.796+0000 7f1bf259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b406c6f0 0x7f42b413a220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f42a40089b0 tx=0x7f42a40082d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9977a5cc97ed91 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 1 -- 172.21.15.115:0/2136843862 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c013a90 con 0x7fea9c06ec40 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: dump: 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.845 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 1 -- 172.21.15.115:0/2136843862 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c013bf0 con 0x7fea9c06ec40 2023-09-30T01:03:36.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 1 -- 172.21.15.115:0/2136843862 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00b610 con 0x7fea9c06ec40 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa0d98700 10 monclient: dump: 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.847 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 -- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf413b900 msgr2=0x7f1bf40a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf413b900 0x7f1bf40a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 -- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bf40b0c50 msgr2=0x7f1bf4152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bf40b0c50 0x7f1bf4152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 -- 172.21.15.115:0/3807182329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec009a20 con 0x7f1bf40b1020 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bf259c700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406c2d0 tx=0x7f1be4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bdaffd700 10 client.?.objecter ms_handle_connect 0x7f1bf40b1020 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f1bdaffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4000d10 con 0x7f1bf40b1020 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: dump: 2023-09-30T01:03:36.849 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be400bbe0 con 0x7f1bf40b1020 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be401a660 con 0x7f1bf40b1020 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bdaffd700 10 monclient: dump: 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.851 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 0x7f6ff006d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff00644b0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ff00ad0d0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ff00ad4c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006de30 con 0x7f6ff00ad0d0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006d080 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00ad8f0 con 0x7f6ff00644b0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff7ea6700 10 monclient(hunting): authenticate will time out at 1078.247313s 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff6443700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6ff00ad4c0 0x7f6ff006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.115:56260/0 (socket says 172.21.15.115:56260) 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff6443700 1 -- 172.21.15.115:0/2202996596 learned_addr learned my addr 172.21.15.115:0/2202996596 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff5c42700 1 --2- 172.21.15.115:0/2202996596 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 0x7f6ff006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff5c42700 10 monclient(hunting): get_auth_request con 0x7f6ff00644b0 auth_method 0 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff5c42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff5c42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7f6ff5c42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa3ffe700 5 monclient: authenticate success, global_id 4339 2023-09-30T01:03:36.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa3ffe700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.797+0000 7feaa3ffe700 4 set_mon_vals no callback set 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 10 monclient: shutdown 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 -- 172.21.15.115:0/2136843862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c06ec40 msgr2=0x7fea9c13a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/2136843862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c06ec40 0x7fea9c13a3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009320 tx=0x7fea8c009ae0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 -- 172.21.15.115:0/2136843862 shutdown_connections 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/2136843862 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c13a8f0 0x7fea9c13ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/2136843862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c06ec40 0x7fea9c13a3b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 -- 172.21.15.115:0/2136843862 >> 172.21.15.115:0/2136843862 conn(0x7fea9c067bd0 msgr2=0x7fea9c0b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7feaa3ffe700 1 -- 172.21.15.115:0/2136843862 shutdown_connections 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf40073d20 con 0x7fdf4c06e400 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fdf48ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fdf48ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bf91ed700 5 monclient: authenticate success, global_id 4345 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bf91ed700 10 monclient: _renew_subs 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bf91ed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be40094f0 con 0x7f1bf40b1020 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f1bf0d99700 4 set_mon_vals no callback set 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 1 -- 172.21.15.115:0/2202996596 >> v1:172.21.15.157:6789/0 conn(0x7f6ff00ad4c0 legacy=0x7f6ff006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 1 -- 172.21.15.115:0/2628015471 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00b860 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: dump: 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.858 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 1 -- 172.21.15.115:0/2202996596 >> v1:172.21.15.115:6789/0 conn(0x7f6ff00ad0d0 legacy=0x7f6ff006e2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:36.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 1 -- 172.21.15.115:0/2628015471 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac00b9c0 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 1 -- 172.21.15.115:0/2628015471 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00bd60 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecb9ffb700 10 monclient: dump: 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.860 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f1bf91ed700 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f1bf91ed700 10 monclient: _renew_subs 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f1bf91ed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf40a87c0 con 0x7f1bf40b1020 2023-09-30T01:03:36.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f1bf91ed700 1 librados: init done 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 1 -- 172.21.15.115:0/1939770717 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a40126e0 con 0x7f42b406c6f0 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 10 monclient: dump: 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.862 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.798+0000 7f42abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 1 -- 172.21.15.115:0/1939770717 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4012840 con 0x7f42b406c6f0 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 1 -- 172.21.15.115:0/1939770717 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4012bc0 con 0x7f42b406c6f0 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42abfff700 10 monclient: dump: 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.864 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 1 -- 172.21.15.115:0/2202996596 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0140c60 con 0x7f6ff00644b0 2023-09-30T01:03:36.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecc1977700 5 monclient: authenticate success, global_id 4317 2023-09-30T01:03:36.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecc1977700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7fecc1977700 4 set_mon_vals no callback set 2023-09-30T01:03:36.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f6ff5c42700 1 --2- 172.21.15.115:0/2202996596 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 0x7f6ff006d780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009e90 tx=0x7f6fe8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5d9d8a50aecdda server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42ba66e700 5 monclient: authenticate success, global_id 4320 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42ba66e700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.799+0000 7f42ba66e700 4 set_mon_vals no callback set 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 10 monclient: shutdown 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 -- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b406c6f0 msgr2=0x7f42b413a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 --2- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b406c6f0 0x7f42b413a220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f42a40089b0 tx=0x7f42a40082d0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 -- 172.21.15.115:0/1939770717 shutdown_connections 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 --2- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b413a760 0x7f42b413ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 --2- 172.21.15.115:0/1939770717 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b406c6f0 0x7f42b413a220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f42ba66e700 1 -- 172.21.15.115:0/1939770717 >> 172.21.15.115:0/1939770717 conn(0x7f42b40679e0 msgr2=0x7f42b4067de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 10 monclient: shutdown 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 -- 172.21.15.115:0/2628015471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0aaeb0 msgr2=0x7fecbc0ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 --2- 172.21.15.115:0/2628015471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0ac020 tx=0x7fecac009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 1 -- 172.21.15.115:0/2202996596 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe800e070 con 0x7f6ff00644b0 2023-09-30T01:03:36.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: dump: 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.869 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 1 -- 172.21.15.115:0/2202996596 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe800bbf0 con 0x7f6ff00644b0 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 1 -- 172.21.15.115:0/2202996596 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8012430 con 0x7f6ff00644b0 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f6ff4c40700 10 monclient: dump: 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.870 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f1bf91ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f1bf91ed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f1bf91ed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf40a6480 con 0x7f1bf40b1020 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 -- 172.21.15.115:0/2628015471 shutdown_connections 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 --2- 172.21.15.115:0/2628015471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0aaeb0 0x7fecbc0ab2c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 --2- 172.21.15.115:0/2628015471 >> v2:172.21.15.157:3300/0 conn(0x7fecbc0b0e10 0x7fecbc142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fecc1977700 1 -- 172.21.15.115:0/2628015471 >> 172.21.15.115:0/2628015471 conn(0x7fecbc067bd0 msgr2=0x7fecbc0b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fa97e50a700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fa97e50a700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7fa97e50a700 10 monclient: monmap: 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.801011+0000 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.801011+0000 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.873 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.800+0000 7feaa3ffe700 1 -- 172.21.15.115:0/2136843862 wait complete. 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 10 monclient: monmap: 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.875 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7fecc1977700 1 -- 172.21.15.115:0/2628015471 shutdown_connections 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 librados: starting msgr at 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 librados: starting objecter 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 Processor -- start 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 -- start start 2023-09-30T01:03:36.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 librados: setting wanted keys 2023-09-30T01:03:36.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fa97e50a700 1 Processor -- start 2023-09-30T01:03:36.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7fecc1977700 1 -- 172.21.15.115:0/2628015471 wait complete. 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7fecc1977700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7fecc1977700 10 monclient: monmap: 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.878 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 librados: starting msgr at 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 librados: starting objecter 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 Processor -- start 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 -- start start 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 librados: setting wanted keys 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 librados: calling monclient init 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient: init 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:36.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-b con 0x7fecbc06b280 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 0x7fecbc15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-c con 0x7fecbc0aaeb0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0b0e10 0x7fecbc15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): picked mon.noname-a con 0x7fecbc0b0e10 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0a87c0 con 0x7fecbc0b0e10 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0a8c10 con 0x7fecbc06b280 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0a8930 con 0x7fecbc0aaeb0 2023-09-30T01:03:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fecc1977700 10 monclient(hunting): authenticate will time out at 1078.252989s 2023-09-30T01:03:36.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fa97e50a700 1 -- start start 2023-09-30T01:03:36.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.802+0000 7fa97e50a700 10 monclient: init 2023-09-30T01:03:36.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 0x7fecbc15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45532/0 (socket says 172.21.15.115:45532) 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 0x7fecbc15a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59138/0 (socket says 172.21.15.115:59138) 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 -- 172.21.15.115:0/2500760367 learned_addr learned my addr 172.21.15.115:0/2500760367 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): get_auth_request con 0x7fecbc06b280 auth_method 0 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): get_auth_request con 0x7fecbc0aaeb0 auth_method 0 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 1 librados: calling monclient init 2023-09-30T01:03:36.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.801+0000 7feaa3ffe700 10 monclient: init 2023-09-30T01:03:36.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T01:03:36.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa97806e400 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806ec40 0x7fa97813a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-b con 0x7fa97806ec40 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa97813a8f0 0x7fa97813ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-a con 0x7fa97813a8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97806bb30 con 0x7fa97806e400 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97813b270 con 0x7fa97813a8f0 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97806d840 con 0x7fa97806ec40 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fa97e50a700 10 monclient(hunting): authenticate will time out at 1078.253339s 2023-09-30T01:03:36.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-09-30T01:03:36.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 -- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 msgr2=0x7fecbc15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 0x7fecbc15a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 -- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0b0e10 msgr2=0x7fecbc15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0b0e10 0x7fecbc15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 -- 172.21.15.115:0/2500760367 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0094f0 con 0x7fecbc06b280 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7fecbaffd700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecb000d890 tx=0x7fecb0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feca2ffd700 10 client.?.objecter ms_handle_connect 0x7fecbc06b280 2023-09-30T01:03:36.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feca2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feaa3ffe700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feaa3ffe700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feaa3ffe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c06e400 0x7fea9c13c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.803+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-c con 0x7fea9c06e400 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea9c06ec40 0x7fea9c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-a con 0x7fea9c06ec40 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): picked mon.noname-b con 0x7fea9c13a8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c13d190 con 0x7fea9c06ec40 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0a59c0 con 0x7fea9c13a8f0 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0a5c70 con 0x7fea9c06e400 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa3ffe700 10 monclient(hunting): authenticate will time out at 1078.254068s 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45544/0 (socket says 172.21.15.115:45544) 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 -- 172.21.15.115:0/1154813346 learned_addr learned my addr 172.21.15.115:0/1154813346 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): get_auth_request con 0x7fea9c13a8f0 auth_method 0 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1d9a700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c06e400 0x7fea9c13c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1d9a700 10 monclient(hunting): get_auth_request con 0x7fea9c06e400 auth_method 0 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1d9a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1d9a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1d9a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1599700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea9c06ec40 0x7fea9c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1599700 10 monclient(hunting): get_auth_request con 0x7fea9c06ec40 auth_method 0 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1599700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1599700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa1599700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 -- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c06e400 msgr2=0x7fea9c13c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c06e400 0x7fea9c13c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 -- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea9c06ec40 msgr2=0x7fea9c0a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea9c06ec40 0x7fea9c0a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 -- 172.21.15.115:0/1154813346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c009d80 con 0x7fea9c13a8f0 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7feaa259b700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea9400bc30 tx=0x7fea94009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fea91ffb700 10 client.?.objecter ms_handle_connect 0x7fea9c13a8f0 2023-09-30T01:03:36.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fea91ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa97813a8f0 0x7fa97813ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa977fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa97806e400 0x7fa97813d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46662/0 (socket says 172.21.15.115:46662) 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa977fff700 1 -- 172.21.15.115:0/3525025662 learned_addr learned my addr 172.21.15.115:0/3525025662 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): get_auth_request con 0x7fa97813a8f0 auth_method 0 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 -- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806ec40 msgr2=0x7fa97813a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806ec40 0x7fa97813a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 -- 172.21.15.115:0/3525025662 >> v1:172.21.15.115:6789/0 conn(0x7fa97806e400 legacy=0x7fa97813d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7fa97caa7700 1 -- 172.21.15.115:0/3525025662 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97813dde0 con 0x7fa97813a8f0 2023-09-30T01:03:36.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa97813ad30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa96c010c90 tx=0x7fa96c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6156a6597bf605 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7f6ff7ea6700 5 monclient: authenticate success, global_id 4259 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7f6ff7ea6700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.804+0000 7f6ff7ea6700 4 set_mon_vals no callback set 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 10 monclient: shutdown 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/2202996596 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 msgr2=0x7f6ff006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/2202996596 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 0x7f6ff006d780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009e90 tx=0x7f6fe8009780 comp rx=0 tx=0).stop 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/2202996596 shutdown_connections 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/2202996596 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00644b0 0x7f6ff006d780 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/2202996596 >> 172.21.15.115:0/2202996596 conn(0x7f6ff00699e0 msgr2=0x7f6ff0069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f42ba66e700 1 -- 172.21.15.115:0/1939770717 shutdown_connections 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f42ba66e700 1 -- 172.21.15.115:0/1939770717 wait complete. 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f42ba66e700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f42ba66e700 10 monclient: monmap: 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.897 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f42ba66e700 1 librados: starting msgr at 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f42ba66e700 1 librados: starting objecter 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f42ba66e700 1 Processor -- start 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea940092e0 con 0x7fea9c13a8f0 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: dump: 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9400c7a0 con 0x7fea9c13a8f0 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9400cb60 con 0x7fea9c13a8f0 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fea91ffb700 10 monclient: dump: 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.900 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 5 monclient: authenticate success, global_id 4354 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 10 monclient: _renew_subs 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9c138c30 con 0x7fea9c13a8f0 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 10 monclient: _renew_subs 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c1558b0 con 0x7fea9c13a8f0 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7feaa3ffe700 1 librados: init done 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f1be401aa50 con 0x7f1bf40b1020 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 1 --2- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 0x7f1bdc06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1be4085a80 con 0x7f1bf40b1020 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 10 client.4345.objecter ms_dispatch 0x7f1bf4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 3 client.4345.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 3 client.4345.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bf2d9d700 1 --2- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 0x7f1bdc06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bf2d9d700 10 monclient: get_auth_request con 0x7f1bdc0689f0 auth_method 0 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f1bdaffd700 20 client.4345.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1be4053380 con 0x7f1bf40b1020 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f1bdaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f1bdaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 1 -- 172.21.15.115:0/3525025662 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c0100d0 con 0x7fa97813a8f0 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: dump: 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 1 -- 172.21.15.115:0/3525025662 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c010230 con 0x7fa97813a8f0 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 1 -- 172.21.15.115:0/3525025662 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c0105f0 con 0x7fa97813a8f0 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7fa976ffd700 10 monclient: dump: 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.905 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fa97e50a700 5 monclient: authenticate success, global_id 4326 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fa97e50a700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fa97e50a700 4 set_mon_vals no callback set 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7fa97e50a700 10 monclient: shutdown 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7fa97e50a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7fa97e50a700 1 -- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 msgr2=0x7fa97813ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa97813ad30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa96c010c90 tx=0x7fa96c0090f0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 -- 172.21.15.115:0/3525025662 shutdown_connections 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa97813ad30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3525025662 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806ec40 0x7fa97813a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 -- 172.21.15.115:0/3525025662 >> 172.21.15.115:0/3525025662 conn(0x7fa978067bd0 msgr2=0x7fa9780b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/2202996596 shutdown_connections 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.805+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/2202996596 wait complete. 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f6ff7ea6700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f6ff7ea6700 10 monclient: monmap: 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.907 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f6ff7ea6700 1 librados: starting msgr at 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7f6ff7ea6700 1 librados: starting objecter 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f6ff7ea6700 1 Processor -- start 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f6ff7ea6700 1 -- start start 2023-09-30T01:03:36.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f6ff7ea6700 1 librados: setting wanted keys 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f6ff7ea6700 1 librados: calling monclient init 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0009610 con 0x7fecbc06b280 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: dump: 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.909 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb0009770 con 0x7fecbc06b280 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0009b30 con 0x7fecbc06b280 2023-09-30T01:03:36.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7feca2ffd700 10 monclient: dump: 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.911 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 5 monclient: authenticate success, global_id 4351 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 10 monclient: _renew_subs 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc15b150 con 0x7fecbc06b280 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 10 monclient: _renew_subs 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc0a7600 con 0x7fecbc06b280 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecc1977700 1 librados: init done 2023-09-30T01:03:36.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.806+0000 7fecb97fa700 4 set_mon_vals no callback set 2023-09-30T01:03:36.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 -- 172.21.15.115:0/3525025662 shutdown_connections 2023-09-30T01:03:36.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 1 -- 172.21.15.115:0/3525025662 wait complete. 2023-09-30T01:03:36.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f1bf2d9d700 1 --2- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 0x7f1bdc06aea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1be8003a80 tx=0x7f1be8007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f1bdaffd700 10 client.4345.objecter ms_handle_connect 0x7f1bdc0689f0 2023-09-30T01:03:36.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.807+0000 7f6ff7ea6700 10 monclient: init 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f42ba66e700 1 -- start start 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f42ba66e700 1 librados: setting wanted keys 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f42ba66e700 1 librados: calling monclient init 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7f42ba66e700 10 monclient: init 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 0x7f42b413b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-c con 0x7f42b406b6f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 0x7f42b415df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-b con 0x7f42b406c6f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b413a760 0x7f42b40a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): picked mon.noname-a con 0x7f42b413a760 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40a5a30 con 0x7f42b413a760 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40a7390 con 0x7f42b406c6f0 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40a7670 con 0x7f42b406b6f0 2023-09-30T01:03:36.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42ba66e700 10 monclient(hunting): authenticate will time out at 1078.259308s 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 0x7f42b413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 0x7f42b413b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59158/0 (socket says 172.21.15.115:59158) 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 1 -- 172.21.15.115:0/2200743628 learned_addr learned my addr 172.21.15.115:0/2200743628 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 10 monclient(hunting): get_auth_request con 0x7f42b406b6f0 auth_method 0 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b966c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 0x7f42b415df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b8e6b700 10 monclient(hunting): get_auth_request con 0x7f42b406c6f0 auth_method 0 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b8e6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b8e6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f42b8e6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b413a760 0x7f42b40a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): get_auth_request con 0x7f42b413a760 auth_method 0 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b9e6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 -- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 msgr2=0x7f42b413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 0x7f42b413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 -- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b413a760 msgr2=0x7f42b40a5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b413a760 0x7f42b40a5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f42b8e6b700 1 -- 172.21.15.115:0/2200743628 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a4008570 con 0x7f42b406c6f0 2023-09-30T01:03:36.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f42b8e6b700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 0x7f42b415df30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42b406a700 tx=0x7f42a401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7feaa3ffe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7feaa3ffe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7feaa3ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c072960 con 0x7fea9c13a8f0 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fea937fe700 4 set_mon_vals no callback set 2023-09-30T01:03:36.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.808+0000 7fa97e50a700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 10 monclient: monmap: 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 librados: starting msgr at 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 librados: starting objecter 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 Processor -- start 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 -- start start 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 librados: setting wanted keys 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 1 librados: calling monclient init 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7fa97e50a700 10 monclient: init 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 0x7fa9780a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-c con 0x7fa97806e400 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa97806ec40 0x7fa97814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-b con 0x7fa97806ec40 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa9780a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): picked mon.noname-a con 0x7fa97813a8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780a8660 con 0x7fa97813a8f0 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780aa050 con 0x7fa97806ec40 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780a5530 con 0x7fa97806e400 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7fa97e50a700 10 monclient(hunting): authenticate will time out at 1078.260743s 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 0x7fa9780a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 0x7fa9780a7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59172/0 (socket says 172.21.15.115:59172) 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 1 -- 172.21.15.115:0/3618957705 learned_addr learned my addr 172.21.15.115:0/3618957705 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 10 monclient(hunting): get_auth_request con 0x7fa97806e400 auth_method 0 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa977fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa9780a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa97806ec40 0x7fa97814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): get_auth_request con 0x7fa97813a8f0 auth_method 0 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): get_auth_request con 0x7fa97806ec40 auth_method 0 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa9777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 -- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 msgr2=0x7fa9780a7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 0x7fa9780a7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 -- 172.21.15.115:0/3618957705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa97806ec40 msgr2=0x7fa97814c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa97806ec40 0x7fa97814c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 -- 172.21.15.115:0/3618957705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c0107d0 con 0x7fa97813a8f0 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa97caa7700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa9780a8af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa96c01e000 tx=0x7fa96c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa95ffff700 10 client.?.objecter ms_handle_connect 0x7fa97813a8f0 2023-09-30T01:03:36.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fa95ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ff00644b0 0x7f6ff00a8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff00644b0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 0x7f6ff00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-c con 0x7f6ff00ad0d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 0x7f6ff00a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff00ad4c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff0146480 con 0x7f6ff00644b0 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00a99b0 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00a9c60 con 0x7f6ff00ad0d0 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff7ea6700 10 monclient(hunting): authenticate will time out at 1078.259257s 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 0x7f6ff00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 0x7f6ff00a5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59166/0 (socket says 172.21.15.115:59166) 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 1 -- 172.21.15.115:0/3062105941 learned_addr learned my addr 172.21.15.115:0/3062105941 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 10 monclient(hunting): get_auth_request con 0x7f6ff00ad0d0 auth_method 0 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5441700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff6443700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 0x7f6ff00a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5c42700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ff00644b0 0x7f6ff00a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7f6ff5c42700 10 monclient(hunting): get_auth_request con 0x7f6ff00644b0 auth_method 0 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): get_auth_request con 0x7f6ff00ad4c0 auth_method 0 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5c42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5441700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5441700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff5441700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 -- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 msgr2=0x7f6ff00a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 0x7f6ff00a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 -- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ff00644b0 msgr2=0x7f6ff00a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ff00644b0 0x7f6ff00a8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 monclient: found mon.noname-b 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7f6ff6443700 1 -- 172.21.15.115:0/3062105941 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009a20 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6ff6443700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 0x7f6ff00a59c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000c440 tx=0x7f6fe0007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 client.?.objecter ms_handle_connect 0x7f6ff00ad4c0 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0005900 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: dump: 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe0005a60 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0005e00 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe7fff700 4 set_mon_vals no callback set 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7f6fe5ffb700 10 monclient: dump: 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.932 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.809+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fecb001e460 con 0x7fecbc06b280 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 1 --2- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 0x7feca806ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fecb0084330 con 0x7fecbc06b280 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 10 client.4351.objecter ms_dispatch 0x7fecbc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 3 client.4351.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 3 client.4351.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.810+0000 7feca2ffd700 20 client.4351.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fecc1977700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fecc1977700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fecc1977700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.811+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c0050b0 con 0x7fecbc06b280 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f42a97fa700 10 client.?.objecter ms_handle_connect 0x7f42b406c6f0 2023-09-30T01:03:36.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f42a97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fea94012070 con 0x7fea9c13a8f0 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 1 --2- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 0x7fea8006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fea940839c0 con 0x7fea9c13a8f0 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 10 client.4354.objecter ms_dispatch 0x7fea9c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 3 client.4354.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 3 client.4354.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fea91ffb700 20 client.4354.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7feaa1d9a700 1 --2- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 0x7fea8006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7feaa1d9a700 10 monclient: get_auth_request con 0x7fea80068950 auth_method 0 2023-09-30T01:03:36.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7feaa1d9a700 1 --2- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 0x7fea8006ae00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea98003870 tx=0x7fea98009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7fea91ffb700 10 client.4354.objecter ms_handle_connect 0x7fea80068950 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c0100d0 con 0x7fa97813a8f0 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: dump: 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.936 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c006e50 con 0x7fa97813a8f0 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c025e50 con 0x7fa97813a8f0 2023-09-30T01:03:36.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa95ffff700 10 monclient: dump: 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.938 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 5 monclient: authenticate success, global_id 4338 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 10 monclient: _renew_subs 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97814d360 con 0x7fa97813a8f0 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 10 monclient: _renew_subs 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9780a9030 con 0x7fa97813a8f0 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa97e50a700 1 librados: init done 2023-09-30T01:03:36.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fa975ffb700 4 set_mon_vals no callback set 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 5 monclient: authenticate success, global_id 4360 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 10 monclient: _renew_subs 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff00a9fa0 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 10 monclient: _renew_subs 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff00a8c60 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7f6ff7ea6700 1 librados: init done 2023-09-30T01:03:36.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7f6ff7ea6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7f6ff7ea6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7f6ff7ea6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff0072960 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fecba7fc700 1 --2- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 0x7feca806ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.812+0000 7fecba7fc700 10 monclient: get_auth_request con 0x7feca8068880 auth_method 0 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7fecba7fc700 1 --2- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 0x7feca806ad30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0a9c00 tx=0x7fecb40016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.813+0000 7feca2ffd700 10 client.4351.objecter ms_handle_connect 0x7feca8068880 2023-09-30T01:03:36.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.814+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fa96c00b040 con 0x7fa97813a8f0 2023-09-30T01:03:36.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 1 --2- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 0x7fa96406ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa96c08dde0 con 0x7fa97813a8f0 2023-09-30T01:03:36.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 10 client.4338.objecter ms_dispatch 0x7fa978001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 3 client.4338.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 3 client.4338.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 20 client.4338.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa977fff700 1 --2- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 0x7fa96406ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa977fff700 10 monclient: get_auth_request con 0x7fa964068950 auth_method 0 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fd618d31700 10 monclient: get_monmap_and_config 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fd618d31700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fd618d31700 10 monclient: monmap: 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:36.816661+0000 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:36.816661+0000 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:36.944 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa95ffff700 10 client.4338.objecter ms_handle_connect 0x7fa964068950 2023-09-30T01:03:36.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.815+0000 7fa977fff700 1 --2- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 0x7fa96406ae00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa970007d60 tx=0x7fa970009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4010070 con 0x7f42b406c6f0 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 10 monclient: dump: 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.946 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecb0051cf0 con 0x7fecbc06b280 2023-09-30T01:03:36.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7feca2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7feca2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 5 monclient: authenticate success, global_id 4357 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 10 monclient: _renew_subs 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b415e990 con 0x7f42b406c6f0 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 10 monclient: _renew_subs 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b40a6640 con 0x7f42b406c6f0 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.816+0000 7f42ba66e700 1 librados: init done 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7f42ba66e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7f42ba66e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7f42ba66e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:36.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b40a5be0 con 0x7f42b406c6f0 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7fd618d31700 1 Processor -- start 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7fd618d31700 1 -- start start 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.817+0000 7fd618d31700 10 monclient: init 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd6140ae8f0 0x7fd6140a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-a con 0x7fd6140ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6140aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:36.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6141415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140a7040 con 0x7fd6141415c0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6140a57f0 con 0x7fd6140aecc0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6140a5480 con 0x7fd6140ae8f0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd618d31700 10 monclient(hunting): authenticate will time out at 1078.268367s 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd6140ae8f0 0x7fd6140a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd6140ae8f0 0x7fd6140a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44216/0 (socket says 172.21.15.115:44216) 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 -- 172.21.15.115:0/344899143 learned_addr learned my addr 172.21.15.115:0/344899143 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): get_auth_request con 0x7fd6140ae8f0 auth_method 0 2023-09-30T01:03:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 -- 172.21.15.115:0/344899143 >> v1:172.21.15.157:6789/0 conn(0x7fd6140aecc0 legacy=0x7fd6140b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 -- 172.21.15.115:0/344899143 >> v1:172.21.15.115:6789/0 conn(0x7fd6141415c0 legacy=0x7fd6140adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 -- 172.21.15.115:0/344899143 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6140a55c0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.818+0000 7fd61259c700 1 --2- 172.21.15.115:0/344899143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd6140a5d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6040099b0 tx=0x7fd604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14240c014490933f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 1 -- 172.21.15.115:0/344899143 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60400b860 con 0x7fd6140ae8f0 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: dump: 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 1 -- 172.21.15.115:0/344899143 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60400b9c0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 1 -- 172.21.15.115:0/344899143 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60400bd60 con 0x7fd6140ae8f0 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd61159a700 10 monclient: dump: 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 5 monclient: authenticate success, global_id 4341 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 4 set_mon_vals no callback set 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 10 monclient: shutdown 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 1 -- 172.21.15.115:0/344899143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 msgr2=0x7fd6140a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.819+0000 7fd618d31700 1 --2- 172.21.15.115:0/344899143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd6140a5d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6040099b0 tx=0x7fd604009250 comp rx=0 tx=0).stop 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 -- 172.21.15.115:0/344899143 shutdown_connections 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 --2- 172.21.15.115:0/344899143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd6140a5d10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 -- 172.21.15.115:0/344899143 >> 172.21.15.115:0/344899143 conn(0x7fd61406a460 msgr2=0x7fd6140630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 -- 172.21.15.115:0/344899143 shutdown_connections 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 -- 172.21.15.115:0/344899143 wait complete. 2023-09-30T01:03:36.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 10 monclient: build_initial_monmap 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 10 monclient: monmap: 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:36.958 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 librados: starting msgr at 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.820+0000 7fd618d31700 1 librados: starting objecter 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 1 Processor -- start 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4011040 con 0x7f42b406c6f0 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4012bc0 con 0x7f42b406c6f0 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42a97fa700 10 monclient: dump: 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7f42ab7fe700 4 set_mon_vals no callback set 2023-09-30T01:03:36.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 1 -- start start 2023-09-30T01:03:36.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 1 librados: setting wanted keys 2023-09-30T01:03:36.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 1 librados: calling monclient init 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 10 monclient: init 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.821+0000 7fd618d31700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-a con 0x7fd6140ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd6140aecc0 0x7fd6140a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-c con 0x7fd6140aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6141415c0 0x7fd61406daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): picked mon.noname-b con 0x7fd6141415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd614140650 con 0x7fd6140ae8f0 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61406dfe0 con 0x7fd6141415c0 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61415af20 con 0x7fd6140aecc0 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd618d31700 10 monclient(hunting): authenticate will time out at 1078.272069s 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44232/0 (socket says 172.21.15.115:44232) 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 -- 172.21.15.115:0/533513649 learned_addr learned my addr 172.21.15.115:0/533513649 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): get_auth_request con 0x7fd6140ae8f0 auth_method 0 2023-09-30T01:03:36.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:36.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:36.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:36.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea94051340 con 0x7fea9c13a8f0 2023-09-30T01:03:36.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fea91ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fea91ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f6fe00072e0 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 1 --2- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 0x7f6fdc06af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6fe0083490 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 10 client.4360.objecter ms_dispatch 0x7f6ff0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 3 client.4360.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 3 client.4360.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6fe5ffb700 20 client.4360.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6ff5c42700 1 --2- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 0x7f6fdc06af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f6ff5c42700 10 monclient: get_auth_request con 0x7f6fdc068a60 auth_method 0 2023-09-30T01:03:36.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f42a4024440 con 0x7f42b406c6f0 2023-09-30T01:03:36.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7f42a97fa700 1 --2- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 0x7f429806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 -- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd6140aecc0 msgr2=0x7fd6140a7790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd6140aecc0 0x7fd6140a7790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 -- 172.21.15.115:0/533513649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6141415c0 msgr2=0x7fd61406daa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6141415c0 0x7fd61406daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 monclient: found mon.noname-a 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:36.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 -- 172.21.15.115:0/533513649 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6040094f0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd61259c700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6140b0920 tx=0x7fd604017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd6027fc700 10 client.?.objecter ms_handle_connect 0x7fd6140ae8f0 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.822+0000 7fd6027fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd604011ea0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: dump: 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.968 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60401a630 con 0x7fd6140ae8f0 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:36.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60401a9f0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd6027fc700 10 monclient: dump: 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:36.970 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.823+0000 7fd610d99700 4 set_mon_vals no callback set 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 5 monclient: authenticate success, global_id 4347 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 10 monclient: _renew_subs 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61415b440 con 0x7fd6140ae8f0 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 10 monclient: _renew_subs 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61415b820 con 0x7fd6140ae8f0 2023-09-30T01:03:36.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7fd618d31700 1 librados: init done 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7f42b966c700 1 --2- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 0x7f429806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.824+0000 7f42b966c700 10 monclient: get_auth_request con 0x7f4298068800 auth_method 0 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7fd618d31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7fd618d31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7fd618d31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61415ad80 con 0x7fd6140ae8f0 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f42a4056d30 con 0x7f42b406c6f0 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 10 client.4357.objecter ms_dispatch 0x7f42b4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 3 client.4357.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 3 client.4357.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42b966c700 1 --2- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 0x7f429806acb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f42b0003810 tx=0x7f42b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 20 client.4357.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f42a97fa700 10 client.4357.objecter ms_handle_connect 0x7f4298068800 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.825+0000 7f6ff5c42700 1 --2- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 0x7f6fdc06af10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009fd0 tx=0x7f6fe8018000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7f6fe5ffb700 10 client.4360.objecter ms_handle_connect 0x7f6fdc068a60 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fe0050e90 con 0x7f6ff00ad4c0 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7f6fe5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7f6fe5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fd6040233e0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.826+0000 7fd6027fc700 1 --2- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 0x7fd5f406ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.827+0000 7fd611d9b700 1 --2- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 0x7fd5f406ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.827+0000 7fd611d9b700 10 monclient: get_auth_request con 0x7fd5f40688d0 auth_method 0 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.830+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f42a4052970 con 0x7f42b406c6f0 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.830+0000 7f42a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.830+0000 7f42a97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.834+0000 7fa97e50a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.834+0000 7fa97e50a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.834+0000 7fa97e50a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:36.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.834+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9580050b0 con 0x7fa97813a8f0 2023-09-30T01:03:36.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.835+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd604056ec0 con 0x7fd6140ae8f0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.835+0000 7fd6027fc700 10 client.4347.objecter ms_dispatch 0x7fd614001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.835+0000 7fd6027fc700 3 client.4347.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.835+0000 7fd6027fc700 3 client.4347.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.835+0000 7fd6027fc700 20 client.4347.objecter dump_active .. 0 homeless 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.838+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd61415ad80 con 0x7fd6140ae8f0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.838+0000 7fd6027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.838+0000 7fd6027fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.838+0000 7fd611d9b700 1 --2- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 0x7fd5f406ad80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd608001050 tx=0x7fd608007ae0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.838+0000 7fd6027fc700 10 client.4347.objecter ms_handle_connect 0x7fd5f40688d0 2023-09-30T01:03:36.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.845+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa96c05b760 con 0x7fa97813a8f0 2023-09-30T01:03:36.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.845+0000 7fa95ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:36.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:36.845+0000 7fa95ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:37.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.160+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f42a4052420 con 0x7f42b406c6f0 2023-09-30T01:03:37.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.166+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdf400737d0 con 0x7fdf4c06e400 2023-09-30T01:03:37.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.160+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fea94050df0 con 0x7fea9c13a8f0 2023-09-30T01:03:37.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.163+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1be401ad00 con 0x7f1bf40b1020 2023-09-30T01:03:37.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.163+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd60408a020 con 0x7fd6140ae8f0 2023-09-30T01:03:37.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.170+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fecb00517a0 con 0x7fecbc06b280 2023-09-30T01:03:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.170+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa96c05b210 con 0x7fa97813a8f0 2023-09-30T01:03:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.170+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6fe0050940 con 0x7f6ff00ad4c0 2023-09-30T01:03:37.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bf91ed700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:37.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bf91ed700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:37.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bf91ed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:37.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1bf4072960 con 0x7f1bf40b1020 2023-09-30T01:03:37.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bdaffd700 1 -- 172.21.15.115:0/3807182329 <== mon.1 v2:172.21.15.157:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1be40531b0 con 0x7f1bf40b1020 2023-09-30T01:03:37.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bdaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:37.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.195+0000 7f1bdaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.198 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:37.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.198+0000 7f1bf91ed700 10 librados: watch_flush enter 2023-09-30T01:03:37.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.199+0000 7f1bf91ed700 10 librados: watch_flush exit 2023-09-30T01:03:37.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.199+0000 7f1bf91ed700 20 client.4345.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.199+0000 7f1bf91ed700 10 client.4345.objecter successfully canceled tick 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 msgr2=0x7f1bdc06aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 0x7f1bdc06aea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1be8003a80 tx=0x7f1be8007ec0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 10 monclient: shutdown 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 msgr2=0x7f1bf40a6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406c2d0 tx=0x7f1be4017000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 shutdown_connections 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> v2:172.21.15.115:6816/115635 conn(0x7f1bdc0689f0 0x7f1bdc06aea0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1be8003a80 tx=0x7f1be8007ec0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1bf413b900 0x7f1bf40a6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1bf40b0c50 0x7f1bf4152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 --2- 172.21.15.115:0/3807182329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bf40b1020 0x7f1bf40a6780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.200+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 >> 172.21.15.115:0/3807182329 conn(0x7f1bf4067bd0 msgr2=0x7f1bf40afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.201+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 shutdown_connections 2023-09-30T01:03:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.201+0000 7f1bf91ed700 1 -- 172.21.15.115:0/3807182329 wait complete. 2023-09-30T01:03:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.202+0000 7f1bf91ed700 1 librados: shutdown 2023-09-30T01:03:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fecc1977700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fecc1977700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fecc1977700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fec9c005360 con 0x7fecbc06b280 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fdf52b70700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fdf52b70700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fdf52b70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.213+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdf2c005360 con 0x7fdf4c06e400 2023-09-30T01:03:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.216+0000 7fdf48ff9700 1 -- 172.21.15.115:0/1989251803 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf40073b50 con 0x7fdf4c06e400 2023-09-30T01:03:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.216+0000 7fdf48ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.216+0000 7fdf48ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.216 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.217+0000 7f6ff7ea6700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.217+0000 7f6ff7ea6700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.217+0000 7f6ff7ea6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.217+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6ff00a9dd0 con 0x7f6ff00ad4c0 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.217+0000 7feca2ffd700 1 -- 172.21.15.115:0/2500760367 <== mon.1 v2:172.21.15.157:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fecb0051b20 con 0x7fecbc06b280 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.218+0000 7feca2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.218+0000 7feca2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 10 librados: watch_flush enter 2023-09-30T01:03:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 10 librados: watch_flush exit 2023-09-30T01:03:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 20 client.4314.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 10 client.4314.objecter successfully canceled tick 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 msgr2=0x7fdf3806ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 0x7fdf3806ae00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c06fa80 tx=0x7fdf44009840 comp rx=0 tx=0).stop 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 10 monclient: shutdown 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 msgr2=0x7fdf4c14c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 0x7fdf4c14c8a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000dd70 tx=0x7fdf40008040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7f6fe5ffb700 1 -- 172.21.15.115:0/3062105941 <== mon.1 v2:172.21.15.157:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6fe0054920 con 0x7f6ff00ad4c0 2023-09-30T01:03:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7f6fe5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7f6fe5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2023-09-30T01:03:37.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 shutdown_connections 2023-09-30T01:03:37.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> v2:172.21.15.115:6816/115635 conn(0x7fdf38068950 0x7fdf3806ae00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf4c13db30 0x7fdf4c0a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdf4c06e400 0x7fdf4c14c8a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 --2- 172.21.15.115:0/1989251803 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdf4c13d6d0 0x7fdf4c0a6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 >> 172.21.15.115:0/1989251803 conn(0x7fdf4c067bd0 msgr2=0x7fdf4c0ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7feaa3ffe700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7feaa3ffe700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7feaa3ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:37.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.219+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fea9c0aab40 con 0x7fea9c13a8f0 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fea91ffb700 1 -- 172.21.15.115:0/1154813346 <== mon.1 v2:172.21.15.157:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea94051170 con 0x7fea9c13a8f0 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fea91ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.220+0000 7fea91ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd618d31700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd618d31700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:37.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd618d31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:37.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd61415ad80 con 0x7fd6140ae8f0 2023-09-30T01:03:37.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fecc1977700 10 librados: watch_flush enter 2023-09-30T01:03:37.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 shutdown_connections 2023-09-30T01:03:37.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fdf52b70700 1 -- 172.21.15.115:0/1989251803 wait complete. 2023-09-30T01:03:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd6027fc700 1 -- 172.21.15.115:0/533513649 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd61415ad80 con 0x7fd6140ae8f0 2023-09-30T01:03:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd6027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fd6027fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.226 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.221+0000 7fdf52b70700 1 librados: shutdown 2023-09-30T01:03:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 10 librados: watch_flush enter 2023-09-30T01:03:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 10 librados: watch_flush exit 2023-09-30T01:03:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 20 client.4351.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 10 client.4351.objecter successfully canceled tick 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 10 librados: watch_flush exit 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 20 client.4360.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 10 client.4360.objecter successfully canceled tick 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 msgr2=0x7f6fdc06af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 0x7f6fdc06af10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009fd0 tx=0x7f6fe8018000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 10 monclient: shutdown 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 msgr2=0x7f6ff00a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 0x7f6ff00a59c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000c440 tx=0x7f6fe0007ea0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 shutdown_connections 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> v2:172.21.15.115:6816/115635 conn(0x7f6fdc068a60 0x7f6fdc06af10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff00ad4c0 0x7f6ff00a59c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ff00644b0 0x7f6ff00a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 --2- 172.21.15.115:0/3062105941 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6ff00ad0d0 0x7f6ff00a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 >> 172.21.15.115:0/3062105941 conn(0x7f6ff00699e0 msgr2=0x7f6ff00b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fd618d31700 10 librados: watch_flush enter 2023-09-30T01:03:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fd618d31700 10 librados: watch_flush exit 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fd618d31700 20 client.4347.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fd618d31700 10 client.4347.objecter successfully canceled tick 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 shutdown_connections 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 msgr2=0x7feca806ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 0x7feca806ad30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0a9c00 tx=0x7fecb40016f0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 10 monclient: shutdown 2023-09-30T01:03:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 msgr2=0x7fecbc0a8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecb000d890 tx=0x7fecb0008040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 shutdown_connections 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> v2:172.21.15.115:6816/115635 conn(0x7feca8068880 0x7feca806ad30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecbc0b0e10 0x7fecbc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecbc06b280 0x7fecbc0a8280 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecb000d890 tx=0x7fecb0008040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 --2- 172.21.15.115:0/2500760367 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fecbc0aaeb0 0x7fecbc15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.223+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 >> 172.21.15.115:0/2500760367 conn(0x7fecbc067bd0 msgr2=0x7fecbc06c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7f6ff7ea6700 1 -- 172.21.15.115:0/3062105941 wait complete. 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 msgr2=0x7fd5f406ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 0x7fd5f406ad80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd608001050 tx=0x7fd608007ae0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 10 monclient: shutdown 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 msgr2=0x7fd61406b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6140b0920 tx=0x7fd604017000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 shutdown_connections 2023-09-30T01:03:37.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fecc1977700 1 -- 172.21.15.115:0/2500760367 wait complete. 2023-09-30T01:03:37.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7fecc1977700 1 librados: shutdown 2023-09-30T01:03:37.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.224+0000 7f6ff7ea6700 1 librados: shutdown 2023-09-30T01:03:37.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 10 librados: watch_flush enter 2023-09-30T01:03:37.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa97e50a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:37.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa97e50a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:37.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa97e50a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:37.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 10 librados: watch_flush exit 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa958005360 con 0x7fa97813a8f0 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 20 client.4354.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 10 client.4354.objecter successfully canceled tick 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 msgr2=0x7fea8006ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 0x7fea8006ae00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea98003870 tx=0x7fea98009040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 10 monclient: shutdown 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 msgr2=0x7fea9c1551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea9400bc30 tx=0x7fea94009840 comp rx=0 tx=0).stop 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 shutdown_connections 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> v2:172.21.15.115:6816/115635 conn(0x7fea80068950 0x7fea8006ae00 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea98003870 tx=0x7fea98009040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9c13a8f0 0x7fea9c1551b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fea9c06e400 0x7fea9c13c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 --2- 172.21.15.115:0/1154813346 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fea9c06ec40 0x7fea9c0a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 >> 172.21.15.115:0/1154813346 conn(0x7fea9c067bd0 msgr2=0x7fea9c06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.236 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T01:03:37.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa95ffff700 1 -- 172.21.15.115:0/3618957705 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa96c05b590 con 0x7fa97813a8f0 2023-09-30T01:03:37.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa95ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:37.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.225+0000 7fa95ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 10 librados: watch_flush enter 2023-09-30T01:03:37.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 shutdown_connections 2023-09-30T01:03:37.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 10 librados: watch_flush exit 2023-09-30T01:03:37.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 20 client.4338.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 10 client.4338.objecter successfully canceled tick 2023-09-30T01:03:37.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7feaa3ffe700 1 -- 172.21.15.115:0/1154813346 wait complete. 2023-09-30T01:03:37.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 msgr2=0x7fa96406ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 0x7fa96406ae00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa970007d60 tx=0x7fa970009040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fa97e50a700 10 monclient: shutdown 2023-09-30T01:03:37.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 msgr2=0x7fa9780a8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa9780a8af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa96c01e000 tx=0x7fa96c01f000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 shutdown_connections 2023-09-30T01:03:37.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> v2:172.21.15.115:6816/115635 conn(0x7fd5f40688d0 0x7fd5f406ad80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6141415c0 0x7fd61406daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd6140ae8f0 0x7fd61406b8f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 --2- 172.21.15.115:0/533513649 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd6140aecc0 0x7fd6140a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.228+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 >> 172.21.15.115:0/533513649 conn(0x7fd61406a460 msgr2=0x7fd614142700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42ba66e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:37.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42ba66e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:37.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42ba66e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:37.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f42b40a5be0 con 0x7f42b406c6f0 2023-09-30T01:03:37.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7feaa3ffe700 1 librados: shutdown 2023-09-30T01:03:37.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 shutdown_connections 2023-09-30T01:03:37.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fd618d31700 1 -- 172.21.15.115:0/533513649 wait complete. 2023-09-30T01:03:37.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 shutdown_connections 2023-09-30T01:03:37.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> v2:172.21.15.115:6816/115635 conn(0x7fa964068950 0x7fa96406ae00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa97813a8f0 0x7fa9780a8af0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa97806e400 0x7fa9780a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 --2- 172.21.15.115:0/3618957705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa97806ec40 0x7fa97814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 >> 172.21.15.115:0/3618957705 conn(0x7fa978067bd0 msgr2=0x7fa978070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 shutdown_connections 2023-09-30T01:03:37.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42a97fa700 1 -- 172.21.15.115:0/2200743628 <== mon.1 v2:172.21.15.157:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42b40a5be0 con 0x7f42b406c6f0 2023-09-30T01:03:37.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42a97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:37.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7f42a97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:37.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.229+0000 7fd618d31700 1 librados: shutdown 2023-09-30T01:03:37.253 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T01:03:37.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.230+0000 7fa97e50a700 1 -- 172.21.15.115:0/3618957705 wait complete. 2023-09-30T01:03:37.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.230+0000 7fa97e50a700 1 librados: shutdown 2023-09-30T01:03:37.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.233+0000 7f42ba66e700 10 librados: watch_flush enter 2023-09-30T01:03:37.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.233+0000 7f42ba66e700 10 librados: watch_flush exit 2023-09-30T01:03:37.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.233+0000 7f42ba66e700 20 client.4357.objecter shutdown clearing up homeless session... 2023-09-30T01:03:37.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.233+0000 7f42ba66e700 10 client.4357.objecter successfully canceled tick 2023-09-30T01:03:37.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 msgr2=0x7f429806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 0x7f429806acb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f42b0003810 tx=0x7f42b0009040 comp rx=0 tx=0).stop 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 10 monclient: shutdown 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 msgr2=0x7f42b415df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 0x7f42b415df30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42b406a700 tx=0x7f42a401b000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 shutdown_connections 2023-09-30T01:03:37.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> v2:172.21.15.115:6816/115635 conn(0x7f4298068800 0x7f429806acb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42b413a760 0x7f42b40a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f42b406b6f0 0x7f42b413b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:37.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 --2- 172.21.15.115:0/2200743628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42b406c6f0 0x7f42b415df30 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42b406a700 tx=0x7f42a401b000 comp rx=0 tx=0).stop 2023-09-30T01:03:37.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.236+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 >> 172.21.15.115:0/2200743628 conn(0x7f42b40679e0 msgr2=0x7f42b413ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:37.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.239+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 shutdown_connections 2023-09-30T01:03:37.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.239+0000 7f42ba66e700 1 -- 172.21.15.115:0/2200743628 wait complete. 2023-09-30T01:03:37.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:37.242+0000 7f42ba66e700 1 librados: shutdown 2023-09-30T01:03:37.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.5 2023-09-30T01:03:37.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.3 2023-09-30T01:03:37.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2023-09-30T01:03:37.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2023-09-30T01:03:37.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-09-30T01:03:37.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.2 2023-09-30T01:03:37.267 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.0 2023-09-30T01:03:38.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T01:03:38.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:03:38.262 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:03:38.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:03:38.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:03:38.266 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T01:03:38.269 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:03:38.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.400+0000 7fceedec4700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.400+0000 7fceedec4700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.400+0000 7fceedec4700 10 monclient: monmap: 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.401745+0000 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.401745+0000 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.401 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.402+0000 7fceedec4700 1 Processor -- start 2023-09-30T01:03:38.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.402+0000 7fceedec4700 1 -- start start 2023-09-30T01:03:38.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.402+0000 7fceedec4700 10 monclient: init 2023-09-30T01:03:38.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcee8069ec0 0x7fcee813d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-c con 0x7fcee8069ec0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcee806b800 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcee806e1f0 0x7fcee806e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-a con 0x7fcee806e1f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee806d810 con 0x7fcee806b800 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee806f690 con 0x7fcee806e1f0 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee806daf0 con 0x7fcee8069ec0 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.403+0000 7fceedec4700 10 monclient(hunting): authenticate will time out at 1079.853816s 2023-09-30T01:03:38.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcee8069ec0 0x7fcee813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcee8069ec0 0x7fcee813d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45566/0 (socket says 172.21.15.115:45566) 2023-09-30T01:03:38.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 1 -- 172.21.15.115:0/2948037187 learned_addr learned my addr 172.21.15.115:0/2948037187 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): get_auth_request con 0x7fcee8069ec0 auth_method 0 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.404+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 -- 172.21.15.115:0/2948037187 >> v2:172.21.15.115:3300/0 conn(0x7fcee806e1f0 msgr2=0x7fcee806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 --2- 172.21.15.115:0/2948037187 >> v2:172.21.15.115:3300/0 conn(0x7fcee806e1f0 0x7fcee806e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 -- 172.21.15.115:0/2948037187 >> v1:172.21.15.115:6789/0 conn(0x7fcee806b800 legacy=0x7fcee806bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:38.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 monclient: found mon.noname-c 2023-09-30T01:03:38.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 -- 172.21.15.115:0/2948037187 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee813e410 con 0x7fcee8069ec0 2023-09-30T01:03:38.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.405+0000 7fcee77fe700 1 --2- 172.21.15.115:0/2948037187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee8069ec0 0x7fcee813d760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcee80acf10 tx=0x7fced8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd1850cd9a649b4c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 1 -- 172.21.15.115:0/2948037187 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800b820 con 0x7fcee8069ec0 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: dump: 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.408 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 1 -- 172.21.15.115:0/2948037187 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced800b980 con 0x7fcee8069ec0 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 1 -- 172.21.15.115:0/2948037187 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800bd20 con 0x7fcee8069ec0 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fcee67fc700 10 monclient: dump: 2023-09-30T01:03:38.409 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fceedec4700 5 monclient: authenticate success, global_id 4366 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fceedec4700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.406+0000 7fceedec4700 4 set_mon_vals no callback set 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 10 monclient: shutdown 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 -- 172.21.15.115:0/2948037187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee8069ec0 msgr2=0x7fcee813d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 --2- 172.21.15.115:0/2948037187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee8069ec0 0x7fcee813d760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcee80acf10 tx=0x7fced8009250 comp rx=0 tx=0).stop 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 -- 172.21.15.115:0/2948037187 shutdown_connections 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 --2- 172.21.15.115:0/2948037187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee8069ec0 0x7fcee813d760 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcee80acf10 tx=0x7fced8009250 comp rx=0 tx=0).stop 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 --2- 172.21.15.115:0/2948037187 >> v2:172.21.15.115:3300/0 conn(0x7fcee806e1f0 0x7fcee806e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.407+0000 7fceedec4700 1 -- 172.21.15.115:0/2948037187 >> 172.21.15.115:0/2948037187 conn(0x7fcee80b0540 msgr2=0x7fcee80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.413+0000 7fceedec4700 1 -- 172.21.15.115:0/2948037187 shutdown_connections 2023-09-30T01:03:38.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.413+0000 7fceedec4700 1 -- 172.21.15.115:0/2948037187 wait complete. 2023-09-30T01:03:38.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.414+0000 7fceedec4700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.414+0000 7fceedec4700 10 monclient: monmap: 2023-09-30T01:03:38.414 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.414 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.414+0000 7fceedec4700 1 librados: starting msgr at 2023-09-30T01:03:38.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.414+0000 7fceedec4700 1 librados: starting objecter 2023-09-30T01:03:38.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.415+0000 7fceedec4700 1 Processor -- start 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.415+0000 7fceedec4700 1 -- start start 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.415+0000 7fceedec4700 1 librados: setting wanted keys 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.415+0000 7fceedec4700 1 librados: calling monclient init 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.415+0000 7fceedec4700 10 monclient: init 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 0x7fcee80a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-a con 0x7fcee806b800 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee806e1f0 0x7fcee80a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-b con 0x7fcee806e1f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 0x7fcee80a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): picked mon.noname-c con 0x7fcee80a61a0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80a9340 con 0x7fcee806b800 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80a9040 con 0x7fcee806e1f0 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee812d040 con 0x7fcee80a61a0 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.416+0000 7fceedec4700 10 monclient(hunting): authenticate will time out at 1079.866899s 2023-09-30T01:03:38.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 0x7fcee80a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 0x7fcee80a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44236/0 (socket says 172.21.15.115:44236) 2023-09-30T01:03:38.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 1 -- 172.21.15.115:0/1081586874 learned_addr learned my addr 172.21.15.115:0/1081586874 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 10 monclient(hunting): get_auth_request con 0x7fcee806b800 auth_method 0 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.417+0000 7fcee77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 0x7fcee80a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): get_auth_request con 0x7fcee80a61a0 auth_method 0 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee6ffd700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee806e1f0 0x7fcee80a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee6ffd700 10 monclient(hunting): get_auth_request con 0x7fcee806e1f0 auth_method 0 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 -- 172.21.15.115:0/1081586874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee806e1f0 msgr2=0x7fcee80a9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee806e1f0 0x7fcee80a9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 -- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 msgr2=0x7fcee80a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 0x7fcee80a9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.418+0000 7fcee7fff700 1 monclient: found mon.noname-c 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcee7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcee7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcee7fff700 1 -- 172.21.15.115:0/1081586874 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80094f0 con 0x7fcee80a61a0 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcee7fff700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 0x7fcee80a6620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcee000bb00 tx=0x7fcee0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 client.?.objecter ms_handle_connect 0x7fcee80a61a0 2023-09-30T01:03:38.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000c840 con 0x7fcee80a61a0 2023-09-30T01:03:38.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:38.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: dump: 2023-09-30T01:03:38.423 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000c9c0 con 0x7fcee80a61a0 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000cd80 con 0x7fcee80a61a0 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.419+0000 7fcecf7fe700 10 monclient: dump: 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.425 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fcee5ffb700 4 set_mon_vals no callback set 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 5 monclient: authenticate success, global_id 4277 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 10 monclient: _renew_subs 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee80a7da0 con 0x7fcee80a61a0 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 10 monclient: _renew_subs 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee80a6d20 con 0x7fcee80a61a0 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.420+0000 7fceedec4700 1 librados: init done 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.422+0000 7fceedec4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.422+0000 7fceedec4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.422+0000 7fceedec4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.422+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee80a9180 con 0x7fcee80a61a0 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcee0014620 con 0x7fcee80a61a0 2023-09-30T01:03:38.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 1 --2- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 0x7fced406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcee0083d80 con 0x7fcee80a61a0 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 10 client.4277.objecter ms_dispatch 0x7fcee8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 3 client.4277.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.423+0000 7fcecf7fe700 3 client.4277.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.424+0000 7fcecf7fe700 20 client.4277.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.424+0000 7fcee77fe700 1 --2- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 0x7fced406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.424+0000 7fcee77fe700 10 monclient: get_auth_request con 0x7fced40689e0 auth_method 0 2023-09-30T01:03:38.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.424+0000 7fcee77fe700 1 --2- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 0x7fced406ae90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fced8006e80 tx=0x7fced8016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.425+0000 7fcecf7fe700 10 client.4277.objecter ms_handle_connect 0x7fced40689e0 2023-09-30T01:03:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.433+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcee004c5a0 con 0x7fcee80a61a0 2023-09-30T01:03:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.433+0000 7fcecf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.433+0000 7fcecf7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.467+0000 7f2bb885e700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.467+0000 7f2bb885e700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.467+0000 7f2bb885e700 10 monclient: monmap: 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.468260+0000 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.468260+0000 2023-09-30T01:03:38.468 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.469 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f2bb885e700 1 Processor -- start 2023-09-30T01:03:38.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f2bb885e700 1 -- start start 2023-09-30T01:03:38.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f2bb885e700 10 monclient: init 2023-09-30T01:03:38.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T01:03:38.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2bb40ae8f0 0x7f2bb40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-a con 0x7f2bb40ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-b con 0x7f2bb40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2bb41415c0 0x7f2bb40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-c con 0x7f2bb41415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb40a7040 con 0x7f2bb40ae8f0 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb40b19b0 con 0x7f2bb41415c0 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb40a57f0 con 0x7f2bb40aecc0 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.471+0000 7f2bb885e700 10 monclient(hunting): authenticate will time out at 1079.921764s 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f25406d8700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f25406d8700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.470+0000 7f25406d8700 10 monclient: monmap: 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.473 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.471411+0000 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.471411+0000 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.474 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59182/0 (socket says 172.21.15.115:59182) 2023-09-30T01:03:38.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3895626044 learned_addr learned my addr 172.21.15.115:0/3895626044 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): get_auth_request con 0x7f2bb40aecc0 auth_method 0 2023-09-30T01:03:38.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3895626044 >> v2:172.21.15.157:3300/0 conn(0x7f2bb41415c0 msgr2=0x7f2bb40adb60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3895626044 >> v2:172.21.15.157:3300/0 conn(0x7f2bb41415c0 0x7f2bb40adb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3895626044 >> v2:172.21.15.115:3300/0 conn(0x7f2bb40ae8f0 msgr2=0x7f2bb40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3895626044 >> v2:172.21.15.115:3300/0 conn(0x7f2bb40ae8f0 0x7f2bb40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 monclient: found mon.noname-b 2023-09-30T01:03:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.472+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3895626044 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40a5480 con 0x7f2bb40aecc0 2023-09-30T01:03:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.473+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3895626044 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40b0340 tx=0x7f2ba4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9e5f8a98abbfd15 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.474+0000 7f25406d8700 1 Processor -- start 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.474+0000 7f7c1548a700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.474+0000 7f7c1548a700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.474+0000 7f7c1548a700 10 monclient: monmap: 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.475776+0000 2023-09-30T01:03:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.475776+0000 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 1 -- 172.21.15.115:0/3895626044 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400b860 con 0x7f2bb40aecc0 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: dump: 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 1 -- 172.21.15.115:0/3895626044 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba400b9c0 con 0x7f2bb40aecc0 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 1 -- 172.21.15.115:0/3895626044 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400bd40 con 0x7f2bb40aecc0 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb159a700 10 monclient: dump: 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f25406d8700 1 -- start start 2023-09-30T01:03:38.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f25406d8700 10 monclient: init 2023-09-30T01:03:38.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb885e700 5 monclient: authenticate success, global_id 4280 2023-09-30T01:03:38.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb885e700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.475+0000 7f2bb885e700 4 set_mon_vals no callback set 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 10 monclient: shutdown 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 -- 172.21.15.115:0/3895626044 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 msgr2=0x7f2bb40b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3895626044 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40b0340 tx=0x7f2ba4009250 comp rx=0 tx=0).stop 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 -- 172.21.15.115:0/3895626044 shutdown_connections 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3895626044 >> v2:172.21.15.115:3300/0 conn(0x7f2bb40ae8f0 0x7f2bb40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3895626044 >> v2:172.21.15.157:3300/0 conn(0x7f2bb41415c0 0x7f2bb40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3895626044 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb40aecc0 0x7f2bb40b1470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 -- 172.21.15.115:0/3895626044 >> 172.21.15.115:0/3895626044 conn(0x7f2bb406a460 msgr2=0x7f2bb40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 -- 172.21.15.115:0/3895626044 shutdown_connections 2023-09-30T01:03:38.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f25380b0c50 0x7f253806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-b con 0x7f25380b0c50 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f253806e400 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f253813a3b0 0x7f253813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-a con 0x7f253813a3b0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253806bb30 con 0x7f253806e400 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253806d840 con 0x7f253813a3b0 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253806e830 con 0x7f25380b0c50 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f25406d8700 10 monclient(hunting): authenticate will time out at 1079.926491s 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f253dc73700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f253806e400 0x7f253813d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46676/0 (socket says 172.21.15.115:46676) 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f253dc73700 1 -- 172.21.15.115:0/3617398185 learned_addr learned my addr 172.21.15.115:0/3617398185 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357295951 0 0) 0x7f253806bb30 con 0x7f253806e400 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 10 monclient(hunting): my global_id is 4353 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2524003570 con 0x7f253806e400 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 --2- 172.21.15.115:0/3617398185 >> v2:172.21.15.115:3300/0 conn(0x7f253813a3b0 0x7f253813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): get_auth_request con 0x7f253813a3b0 auth_method 0 2023-09-30T01:03:38.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 -- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f25380b0c50 msgr2=0x7f253806ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 --2- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f25380b0c50 0x7f253806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 -- 172.21.15.115:0/3617398185 >> v1:172.21.15.115:6789/0 conn(0x7f253806e400 legacy=0x7f253813d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475259916 0 0) 0x7f2524003570 con 0x7f253806e400 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253ec75700 1 -- 172.21.15.115:0/3617398185 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253813dde0 con 0x7f253813a3b0 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f253d472700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.476+0000 7f2bb885e700 1 -- 172.21.15.115:0/3895626044 wait complete. 2023-09-30T01:03:38.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f2bb885e700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f2bb885e700 10 monclient: monmap: 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.490 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f2bb885e700 1 librados: starting msgr at 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f2bb885e700 1 librados: starting objecter 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.477+0000 7f2bb885e700 1 Processor -- start 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253ec75700 1 --2- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253813a3b0 0x7f253813a7f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2530009e60 tx=0x7f2530009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79168d650ae152a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000e070 con 0x7f253813a3b0 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: dump: 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.492 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253000b850 con 0x7f253813a3b0 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 1 -- 172.21.15.115:0/3617398185 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534001470 con 0x7f253813a3b0 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f253d472700 10 monclient: dump: 2023-09-30T01:03:38.493 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.494 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 -- start start 2023-09-30T01:03:38.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 librados: setting wanted keys 2023-09-30T01:03:38.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 librados: calling monclient init 2023-09-30T01:03:38.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient: init 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2bb40ae8f0 0x7f2bb406be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-b con 0x7f2bb40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-a con 0x7f2bb40aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb41415c0 0x7f2bb406c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): picked mon.noname-c con 0x7f2bb41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb413e6e0 con 0x7f2bb40aecc0 2023-09-30T01:03:38.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb406dc20 con 0x7f2bb40ae8f0 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb406df00 con 0x7f2bb41415c0 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb885e700 10 monclient(hunting): authenticate will time out at 1079.928927s 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44262/0 (socket says 172.21.15.115:44262) 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3852377059 learned_addr learned my addr 172.21.15.115:0/3852377059 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): get_auth_request con 0x7f2bb40aecc0 auth_method 0 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb2d9d700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb41415c0 0x7f2bb406c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb2d9d700 10 monclient(hunting): get_auth_request con 0x7f2bb41415c0 auth_method 0 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb41415c0 msgr2=0x7f2bb406c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb41415c0 0x7f2bb406c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3852377059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2bb40ae8f0 msgr2=0x7f2bb406be30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2bb40ae8f0 0x7f2bb406be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 -- 172.21.15.115:0/3852377059 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40094f0 con 0x7f2bb40aecc0 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7fef7ad60700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7fef7ad60700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7fef7ad60700 10 monclient: monmap: 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.479246+0000 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.479246+0000 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.500 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2bb1d9b700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40090b0 tx=0x7f2ba4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 client.?.objecter ms_handle_connect 0x7f2bb40aecc0 2023-09-30T01:03:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f7c1548a700 1 Processor -- start 2023-09-30T01:03:38.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f7c1548a700 1 -- start start 2023-09-30T01:03:38.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f7c1548a700 10 monclient: init 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011e60 con 0x7f2bb40aecc0 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: dump: 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.503 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba401f070 con 0x7f2bb40aecc0 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba401a890 con 0x7f2bb40aecc0 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba27fc700 10 monclient: dump: 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.479+0000 7f2ba3fff700 4 set_mon_vals no callback set 2023-09-30T01:03:38.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 5 monclient: authenticate success, global_id 4359 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 10 monclient: _renew_subs 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb406abf0 con 0x7f2bb40aecc0 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 10 monclient: _renew_subs 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb406c940 con 0x7f2bb40aecc0 2023-09-30T01:03:38.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f2bb885e700 1 librados: init done 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c100ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c100aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7c100a5d10 0x7f7c100adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-a con 0x7f7c100a5d10 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100a6190 con 0x7f7c100ae8f0 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100a7040 con 0x7f7c100aecc0 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c10141cd0 con 0x7f7c100a5d10 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.480+0000 7f7c1548a700 10 monclient(hunting): authenticate will time out at 1079.930831s 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f9f3d883700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f9f3d883700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.478+0000 7f9f3d883700 10 monclient: monmap: 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.508 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.479845+0000 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.479845+0000 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.509 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7c100a5d10 0x7f7c100adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0effd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f7c100ae8f0 0x7f7c100b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46686/0 (socket says 172.21.15.115:46686) 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0effd700 1 -- 172.21.15.115:0/3778250807 learned_addr learned my addr 172.21.15.115:0/3778250807 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): get_auth_request con 0x7f7c100a5d10 auth_method 0 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f9f3d883700 1 Processor -- start 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f2bb885e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f2bb885e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f2bb885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb406dd60 con 0x7f2bb40aecc0 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f9f3d883700 1 -- start start 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f9f3d883700 10 monclient: init 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 -- 172.21.15.115:0/3778250807 >> v1:172.21.15.157:6789/0 conn(0x7f7c100aecc0 legacy=0x7f7c101415c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:38.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 -- 172.21.15.115:0/3778250807 >> v1:172.21.15.115:6789/0 conn(0x7f7c100ae8f0 legacy=0x7f7c100b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 -- 172.21.15.115:0/3778250807 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100a57f0 con 0x7f7c100a5d10 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f25406d8700 5 monclient: authenticate success, global_id 4356 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f25406d8700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f25406d8700 4 set_mon_vals no callback set 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.481+0000 7f7c0f7fe700 1 --2- 172.21.15.115:0/3778250807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100a5d10 0x7f7c100adb60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e60 tx=0x7f7c08009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fe2c5cda2d7180 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 10 monclient: shutdown 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 -- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253813a3b0 msgr2=0x7f253813a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 --2- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253813a3b0 0x7f253813a7f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2530009e60 tx=0x7f2530009780 comp rx=0 tx=0).stop 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 -- 172.21.15.115:0/3617398185 shutdown_connections 2023-09-30T01:03:38.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 --2- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f25380b0c50 0x7f253806ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 --2- 172.21.15.115:0/3617398185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253813a3b0 0x7f253813a7f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f25406d8700 1 -- 172.21.15.115:0/3617398185 >> 172.21.15.115:0/3617398185 conn(0x7f2538067bd0 msgr2=0x7f25380b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 1 -- 172.21.15.115:0/3617398185 shutdown_connections 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 1 -- 172.21.15.115:0/3617398185 wait complete. 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 10 monclient: monmap: 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.515 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 1 librados: starting msgr at 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 1 librados: starting objecter 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7f25406d8700 1 Processor -- start 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 Processor -- start 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 -- start start 2023-09-30T01:03:38.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient: init 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fef7406e400 0x7fef7406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-c con 0x7fef7406e400 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef7413d6d0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fef7413db30 0x7fef7406c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-a con 0x7fef7413db30 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7406bb30 con 0x7fef7413d6d0 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7406d840 con 0x7fef7413db30 2023-09-30T01:03:38.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740aab40 con 0x7fef7406e400 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.484+0000 7fef7ad60700 10 monclient(hunting): authenticate will time out at 1079.934972s 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fef7406e400 0x7fef7406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fef7406e400 0x7fef7406ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45612/0 (socket says 172.21.15.115:45612) 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef73fff700 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fef7413d6d0 0x7fef7406bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.115:56286/0 (socket says 172.21.15.115:56286) 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 -- 172.21.15.115:0/458285118 learned_addr learned my addr 172.21.15.115:0/458285118 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): get_auth_request con 0x7fef7406e400 auth_method 0 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 -- 172.21.15.115:0/458285118 >> v2:172.21.15.115:3300/0 conn(0x7fef7413db30 msgr2=0x7fef7406c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 --2- 172.21.15.115:0/458285118 >> v2:172.21.15.115:3300/0 conn(0x7fef7413db30 0x7fef7406c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 -- 172.21.15.115:0/458285118 >> v1:172.21.15.157:6789/0 conn(0x7fef7413d6d0 legacy=0x7fef7406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 monclient: found mon.noname-c 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef737fe700 1 -- 172.21.15.115:0/458285118 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056219239 0 0) 0x7fef7406bb30 con 0x7fef7413d6d0 2023-09-30T01:03:38.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 -- 172.21.15.115:0/458285118 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7406da40 con 0x7fef7406e400 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef737fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7fef78afc700 1 --2- 172.21.15.115:0/458285118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7406e400 0x7fef7406ec40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fef740ae040 tx=0x7fef680082d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6be3d666c6c9fe4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2ba401ac40 con 0x7f2bb40aecc0 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 1 --2- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 0x7f2b9406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2ba4021070 con 0x7f2bb40aecc0 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 10 client.4359.objecter ms_dispatch 0x7f2bb4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 3 client.4359.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 3 client.4359.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f2ba27fc700 20 client.4359.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 0x7f9f3806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-c con 0x7f9f380b0c50 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f3806e140 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f9f3806e5a0 0x7f9f3813a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-a con 0x7f9f3806e5a0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3806bb30 con 0x7f9f3806e140 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3806d840 con 0x7f9f3806e5a0 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3813dde0 con 0x7f9f380b0c50 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f3d883700 10 monclient(hunting): authenticate will time out at 1079.932286s 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 0x7f9f3806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 0x7f9f3806ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45604/0 (socket says 172.21.15.115:45604) 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 1 -- 172.21.15.115:0/1391520585 learned_addr learned my addr 172.21.15.115:0/1391520585 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 10 monclient(hunting): get_auth_request con 0x7f9f380b0c50 auth_method 0 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f36ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7f25406d8700 1 -- start start 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7f25406d8700 1 librados: setting wanted keys 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7f25406d8700 1 librados: calling monclient init 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.485+0000 7f25406d8700 10 monclient: init 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253806e400 0x7f2538151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-a con 0x7f253806e400 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-b con 0x7f25380b0c50 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f253813a3b0 0x7f2538154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): picked mon.noname-c con 0x7f253813a3b0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538151f70 con 0x7f253806e400 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538152220 con 0x7f25380b0c50 2023-09-30T01:03:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380a9870 con 0x7f253813a3b0 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f25406d8700 10 monclient(hunting): authenticate will time out at 1079.936495s 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 1 -- 172.21.15.115:0/458285118 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68010070 con 0x7fef7406e400 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: dump: 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.525 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 1 -- 172.21.15.115:0/458285118 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6800db00 con 0x7fef7406e400 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 1 -- 172.21.15.115:0/458285118 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6800dea0 con 0x7fef7406e400 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef737fe700 10 monclient: dump: 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.527 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef7ad60700 5 monclient: authenticate success, global_id 4375 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef7ad60700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7fef7ad60700 4 set_mon_vals no callback set 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f253813a3b0 0x7f2538154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:45614/0 (socket says 172.21.15.115:45614) 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 1 -- 172.21.15.115:0/133642469 learned_addr learned my addr 172.21.15.115:0/133642469 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): get_auth_request con 0x7f25380b0c50 auth_method 0 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): get_auth_request con 0x7f253813a3b0 auth_method 0 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253dc73700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.486+0000 7f253ec75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 -- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f253813a3b0 msgr2=0x7f2538154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f253813a3b0 0x7f2538154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 -- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253806e400 msgr2=0x7f2538151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253806e400 0x7f2538151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 monclient: found mon.noname-b 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 -- 172.21.15.115:0/133642469 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530009a20 con 0x7f25380b0c50 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253dc73700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f253400cb90 tx=0x7f2534001a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 client.?.objecter ms_handle_connect 0x7f25380b0c50 2023-09-30T01:03:38.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7fef7ad60700 10 monclient: shutdown 2023-09-30T01:03:38.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7fef7ad60700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7fef7ad60700 1 -- 172.21.15.115:0/458285118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7406e400 msgr2=0x7fef7406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7fef7ad60700 1 --2- 172.21.15.115:0/458285118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7406e400 0x7fef7406ec40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fef740ae040 tx=0x7fef680082d0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534001d10 con 0x7f25380b0c50 2023-09-30T01:03:38.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:38.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: dump: 2023-09-30T01:03:38.533 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2534007360 con 0x7f25380b0c50 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534007700 con 0x7f25380b0c50 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f252e7fc700 10 monclient: dump: 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.535 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f253cc71700 4 set_mon_vals no callback set 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 5 monclient: authenticate success, global_id 4381 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 10 monclient: _renew_subs 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253813a0e0 con 0x7f25380b0c50 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 10 monclient: _renew_subs 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2538152680 con 0x7f25380b0c50 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.487+0000 7f25406d8700 1 librados: init done 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.488+0000 7f25406d8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.488+0000 7f25406d8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.488+0000 7f25406d8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.488+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25381524d0 con 0x7f25380b0c50 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2534004a40 con 0x7f25380b0c50 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 1 --2- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 0x7f251c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f25340835f0 con 0x7f25380b0c50 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 10 client.4381.objecter ms_dispatch 0x7f2538001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 3 client.4381.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 3 client.4381.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.489+0000 7f252e7fc700 20 client.4381.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f377fe700 1 --2- 172.21.15.115:0/1391520585 >> v2:172.21.15.115:3300/0 conn(0x7f9f3806e5a0 0x7f9f3813a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f377fe700 10 monclient(hunting): get_auth_request con 0x7f9f3806e5a0 auth_method 0 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f377fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.482+0000 7f9f377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 -- 172.21.15.115:0/1391520585 >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 msgr2=0x7f9f3806ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 --2- 172.21.15.115:0/1391520585 >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 0x7f9f3806ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 -- 172.21.15.115:0/1391520585 >> v1:172.21.15.115:6789/0 conn(0x7f9f3806e140 legacy=0x7f9f3813d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 -- 172.21.15.115:0/1391520585 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3806da40 con 0x7f9f3806e5a0 2023-09-30T01:03:38.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.483+0000 7f9f377fe700 1 --2- 172.21.15.115:0/1391520585 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f3806e5a0 0x7f9f3813a3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9f380ae040 tx=0x7f9f30009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b30e3de20be806d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ba4053710 con 0x7f2bb40aecc0 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f2ba27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f2ba27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f253e474700 1 --2- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 0x7f251c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f253e474700 10 monclient: get_auth_request con 0x7f251c068960 auth_method 0 2023-09-30T01:03:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f2bb259c700 1 --2- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 0x7f2b9406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f2bb259c700 10 monclient: get_auth_request con 0x7f2b94068960 auth_method 0 2023-09-30T01:03:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f253e474700 1 --2- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 0x7f251c06ae10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2528006fd0 tx=0x7f2528006db0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f252e7fc700 10 client.4381.objecter ms_handle_connect 0x7f251c068960 2023-09-30T01:03:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f2ba27fc700 10 client.4359.objecter ms_handle_connect 0x7f2b94068960 2023-09-30T01:03:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f2bb259c700 1 --2- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 0x7f2b9406ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8006d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 1 -- 172.21.15.115:0/1391520585 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000e070 con 0x7f9f3806e5a0 2023-09-30T01:03:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: dump: 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 1 -- 172.21.15.115:0/1391520585 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3000bc30 con 0x7f9f3806e5a0 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 1 -- 172.21.15.115:0/1391520585 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30012430 con 0x7f9f3806e5a0 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.491+0000 7f9f35ffb700 10 monclient: dump: 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 5 monclient: authenticate success, global_id 4368 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 4 set_mon_vals no callback set 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 -- 172.21.15.115:0/458285118 shutdown_connections 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 --2- 172.21.15.115:0/458285118 >> v2:172.21.15.115:3300/0 conn(0x7fef7413db30 0x7fef7406c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 --2- 172.21.15.115:0/458285118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7406e400 0x7fef7406ec40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 -- 172.21.15.115:0/458285118 >> 172.21.15.115:0/458285118 conn(0x7fef74067bd0 msgr2=0x7fef740b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 -- 172.21.15.115:0/458285118 shutdown_connections 2023-09-30T01:03:38.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7fef7ad60700 1 -- 172.21.15.115:0/458285118 wait complete. 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 10 monclient: shutdown 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 1 -- 172.21.15.115:0/1391520585 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f3806e5a0 msgr2=0x7f9f3813a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.492+0000 7f9f3d883700 1 --2- 172.21.15.115:0/1391520585 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f3806e5a0 0x7f9f3813a3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9f380ae040 tx=0x7f9f30009780 comp rx=0 tx=0).stop 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 -- 172.21.15.115:0/1391520585 shutdown_connections 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 --2- 172.21.15.115:0/1391520585 >> v2:172.21.15.157:3300/0 conn(0x7f9f380b0c50 0x7f9f3806ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 --2- 172.21.15.115:0/1391520585 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f3806e5a0 0x7f9f3813a3b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 -- 172.21.15.115:0/1391520585 >> 172.21.15.115:0/1391520585 conn(0x7f9f38067bd0 msgr2=0x7f9f380b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 1 -- 172.21.15.115:0/3778250807 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0800e070 con 0x7f7c100a5d10 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: dump: 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.548 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 1 -- 172.21.15.115:0/3778250807 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0800bbf0 con 0x7f7c100a5d10 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 1 -- 172.21.15.115:0/3778250807 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08012430 con 0x7f7c100a5d10 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c0dffb700 10 monclient: dump: 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.550 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 -- 172.21.15.115:0/1391520585 shutdown_connections 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f9f3d883700 1 -- 172.21.15.115:0/1391520585 wait complete. 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f1035199700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f1035199700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f1035199700 10 monclient: monmap: 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.494817+0000 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.494817+0000 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.552 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c1548a700 5 monclient: authenticate success, global_id 4362 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c1548a700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7f7c1548a700 4 set_mon_vals no callback set 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 10 monclient: shutdown 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 -- 172.21.15.115:0/3778250807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100a5d10 msgr2=0x7f7c100adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 --2- 172.21.15.115:0/3778250807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100a5d10 0x7f7c100adb60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e60 tx=0x7f7c08009780 comp rx=0 tx=0).stop 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7fef7ad60700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.493+0000 7fef7ad60700 10 monclient: monmap: 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7fef7ad60700 1 librados: starting msgr at 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7fef7ad60700 1 librados: starting objecter 2023-09-30T01:03:38.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7fef7ad60700 1 Processor -- start 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 -- 172.21.15.115:0/3778250807 shutdown_connections 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 --2- 172.21.15.115:0/3778250807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100a5d10 0x7f7c100adb60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 -- 172.21.15.115:0/3778250807 >> 172.21.15.115:0/3778250807 conn(0x7f7c1006a460 msgr2=0x7f7c100630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f9f3d883700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f9f3d883700 10 monclient: monmap: 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.556 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 -- 172.21.15.115:0/3778250807 shutdown_connections 2023-09-30T01:03:38.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2534051010 con 0x7f25380b0c50 2023-09-30T01:03:38.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f252e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f252e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.494+0000 7f7c1548a700 1 -- 172.21.15.115:0/3778250807 wait complete. 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 librados: starting msgr at 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 librados: starting objecter 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 Processor -- start 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 -- start start 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 librados: setting wanted keys 2023-09-30T01:03:38.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 1 librados: calling monclient init 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f9f3d883700 10 monclient: init 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f3806e140 0x7f9f3812ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-b con 0x7f9f3806e140 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9f3806e5a0 0x7f9f3812ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-c con 0x7f9f3806e5a0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): picked mon.noname-a con 0x7f9f380b0c50 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3812b980 con 0x7f9f380b0c50 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f380a6780 con 0x7f9f3806e140 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3812f2d0 con 0x7f9f3806e5a0 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f3d883700 10 monclient(hunting): authenticate will time out at 1079.946148s 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f7c1548a700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.495+0000 7f7c1548a700 10 monclient: monmap: 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f7c1548a700 1 librados: starting msgr at 2023-09-30T01:03:38.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f7c1548a700 1 librados: starting objecter 2023-09-30T01:03:38.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f7c1548a700 1 Processor -- start 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44304/0 (socket says 172.21.15.115:44304) 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 1 -- 172.21.15.115:0/3132610591 learned_addr learned my addr 172.21.15.115:0/3132610591 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): get_auth_request con 0x7f9f380b0c50 auth_method 0 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9f3806e5a0 0x7f9f3812ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): get_auth_request con 0x7f9f3806e5a0 auth_method 0 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 1 -- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9f3806e5a0 msgr2=0x7f9f3812ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.496+0000 7f9f377fe700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9f3806e5a0 0x7f9f3812ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 1 -- 172.21.15.115:0/3132610591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f3806e140 msgr2=0x7f9f3812ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f3806e140 0x7f9f3812ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 1 -- 172.21.15.115:0/3132610591 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30009a20 con 0x7f9f380b0c50 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f377fe700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f380b13d0 tx=0x7f9f30019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 client.?.objecter ms_handle_connect 0x7f9f380b0c50 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30012e50 con 0x7f9f380b0c50 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: dump: 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.567 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f30021070 con 0x7f9f380b0c50 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30026680 con 0x7f9f380b0c50 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f1effd700 10 monclient: dump: 2023-09-30T01:03:38.568 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f357fa700 4 set_mon_vals no callback set 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f7c1548a700 1 -- start start 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f7c1548a700 1 librados: setting wanted keys 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f7c1548a700 1 librados: calling monclient init 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f7c1548a700 10 monclient: init 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c100a5d10 0x7f7c100a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-b con 0x7f7c100a5d10 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-a con 0x7f7c100ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7c100aecc0 0x7f7c10152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): picked mon.noname-c con 0x7f7c100aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c101521a0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100a8690 con 0x7f7c100a5d10 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1006e690 con 0x7f7c100aecc0 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c1548a700 10 monclient(hunting): authenticate will time out at 1079.948142s 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44316/0 (socket says 172.21.15.115:44316) 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 1 -- 172.21.15.115:0/1353246431 learned_addr learned my addr 172.21.15.115:0/1353246431 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): get_auth_request con 0x7f7c100ae8f0 auth_method 0 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 5 monclient: authenticate success, global_id 4371 2023-09-30T01:03:38.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 10 monclient: _renew_subs 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f380a87a0 con 0x7f9f380b0c50 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 10 monclient: _renew_subs 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3812c1f0 con 0x7f9f380b0c50 2023-09-30T01:03:38.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.497+0000 7f9f3d883700 1 librados: init done 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.498+0000 7f7c0e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 -- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7c100aecc0 msgr2=0x7f7c10152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7c100aecc0 0x7f7c10152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 -- 172.21.15.115:0/1353246431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c100a5d10 msgr2=0x7f7c100a7fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c100a5d10 0x7f7c100a7fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 -- 172.21.15.115:0/1353246431 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009a20 con 0x7f7c100ae8f0 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7c0e7fc700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000b5e0 tx=0x7f7c000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7bf6ffd700 10 client.?.objecter ms_handle_connect 0x7f7c100ae8f0 2023-09-30T01:03:38.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f7bf6ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7fef7ad60700 1 -- start start 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7fef7ad60700 1 librados: setting wanted keys 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7fef7ad60700 1 librados: calling monclient init 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7fef7ad60700 10 monclient: init 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-a con 0x7fef7406e400 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef7413d6d0 0x7fef74134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-c con 0x7fef7413d6d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7413db30 0x7fef74135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): picked mon.noname-b con 0x7fef7413db30 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef741365e0 con 0x7fef7406e400 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740a6bd0 con 0x7fef7413db30 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740a6880 con 0x7fef7413d6d0 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef7ad60700 10 monclient(hunting): authenticate will time out at 1079.950186s 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44318/0 (socket says 172.21.15.115:44318) 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 -- 172.21.15.115:0/1953134208 learned_addr learned my addr 172.21.15.115:0/1953134208 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): get_auth_request con 0x7fef7406e400 auth_method 0 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 -- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef7413d6d0 msgr2=0x7fef74134ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:03:38.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef7413d6d0 0x7fef74134ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 -- 172.21.15.115:0/1953134208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7413db30 msgr2=0x7fef74135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7413db30 0x7fef74135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 -- 172.21.15.115:0/1953134208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68008570 con 0x7fef7406e400 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef78afc700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef740aac80 tx=0x7fef6801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef70ff9700 10 client.?.objecter ms_handle_connect 0x7fef7406e400 2023-09-30T01:03:38.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.500+0000 7fef70ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f9f3d883700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f9f3d883700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f9f3d883700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.499+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3812c380 con 0x7f9f380b0c50 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.503+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9f30023030 con 0x7f9f380b0c50 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 1 --2- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 0x7f9f2406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9f30086800 con 0x7f9f380b0c50 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 10 client.4371.objecter ms_dispatch 0x7f9f38001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 3 client.4371.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 3 client.4371.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 20 client.4371.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000f510 con 0x7f7c100ae8f0 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 10 monclient: dump: 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7bf6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f1035199700 1 Processor -- start 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f300540a0 con 0x7f9f380b0c50 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f36ffd700 1 --2- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 0x7f9f2406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f9f1effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.505+0000 7f9f36ffd700 10 monclient: get_auth_request con 0x7f9f24068890 auth_method 0 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 5 monclient: authenticate success, global_id 4374 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 10 monclient: _renew_subs 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c10152b70 con 0x7f7c100ae8f0 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 10 monclient: _renew_subs 2023-09-30T01:03:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1006caf0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.504+0000 7f7c1548a700 1 librados: init done 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.505+0000 7f7c1548a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7f7c1548a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7f7c1548a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c10152ea0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68016b80 con 0x7fef7406e400 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: dump: 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.586 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef68016ce0 con 0x7fef7406e400 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6800b2d0 con 0x7fef7406e400 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef70ff9700 10 monclient: dump: 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7f9f1effd700 10 client.4371.objecter ms_handle_connect 0x7f9f24068890 2023-09-30T01:03:38.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7f9f36ffd700 1 --2- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 0x7f9f2406ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c006d40 tx=0x7f9f2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 5 monclient: authenticate success, global_id 4377 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 10 monclient: _renew_subs 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef740a8930 con 0x7fef7406e400 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 10 monclient: _renew_subs 2023-09-30T01:03:38.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef740a5780 con 0x7fef7406e400 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef7ad60700 1 librados: init done 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.506+0000 7fef72ffd700 4 set_mon_vals no callback set 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fef6800b640 con 0x7fef7406e400 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7fef70ff9700 1 --2- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 0x7fef6006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fef68085150 con 0x7fef7406e400 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef70ff9700 10 client.4377.objecter ms_dispatch 0x7fef74001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef70ff9700 3 client.4377.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef70ff9700 3 client.4377.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0000f880 con 0x7f7c100ae8f0 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0000f9f0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 10 monclient: dump: 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.591 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7c0000fb90 con 0x7f7c100ae8f0 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.507+0000 7f7bf6ffd700 1 --2- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 0x7f7bfc06ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7c00083a30 con 0x7f7c100ae8f0 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7bf6ffd700 10 client.4374.objecter ms_dispatch 0x7f7c10001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7bf6ffd700 3 client.4374.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7bf6ffd700 3 client.4374.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7bf6ffd700 20 client.4374.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.508+0000 7f7c0d7fa700 4 set_mon_vals no callback set 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef70ff9700 20 client.4377.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef73fff700 1 --2- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 0x7fef6006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7fef73fff700 10 monclient: get_auth_request con 0x7fef60068960 auth_method 0 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.510+0000 7fef70ff9700 10 client.4377.objecter ms_handle_connect 0x7fef60068960 2023-09-30T01:03:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.510+0000 7fef73fff700 1 --2- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 0x7fef6006ae10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef7413d1a0 tx=0x7fef6c0016e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7f7c0effd700 1 --2- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 0x7f7bfc06ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.509+0000 7f7c0effd700 10 monclient: get_auth_request con 0x7f7bfc068990 auth_method 0 2023-09-30T01:03:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.510+0000 7f7c0effd700 1 --2- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 0x7f7bfc06ae40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c040058d0 tx=0x7f7c04007ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.510+0000 7f7bf6ffd700 10 client.4374.objecter ms_handle_connect 0x7f7bfc068990 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.511+0000 7f1035199700 1 -- start start 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.511+0000 7f1035199700 10 monclient: init 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f10300ae8f0 0x7f10300a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): picked mon.noname-a con 0x7f10300ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): picked mon.noname-b con 0x7f10300aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10301415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300a7040 con 0x7f10301415c0 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10300b19b0 con 0x7f10300ae8f0 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10300a57f0 con 0x7f10300aecc0 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.512+0000 7f1035199700 10 monclient(hunting): authenticate will time out at 1079.962228s 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59256/0 (socket says 172.21.15.115:59256) 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 -- 172.21.15.115:0/2601229236 learned_addr learned my addr 172.21.15.115:0/2601229236 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): get_auth_request con 0x7f10300aecc0 auth_method 0 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 -- 172.21.15.115:0/2601229236 >> v2:172.21.15.115:3300/0 conn(0x7f10300ae8f0 msgr2=0x7f10300a5d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 --2- 172.21.15.115:0/2601229236 >> v2:172.21.15.115:3300/0 conn(0x7f10300ae8f0 0x7f10300a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 -- 172.21.15.115:0/2601229236 >> v1:172.21.15.157:6789/0 conn(0x7f10301415c0 legacy=0x7f10300adb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:03:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 monclient: found mon.noname-b 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 -- 172.21.15.115:0/2601229236 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300a5480 con 0x7f10300aecc0 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.518+0000 7f102e59c700 1 --2- 172.21.15.115:0/2601229236 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1020009f20 tx=0x7f1020009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2797b3ce12dd92d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c000512c0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f7bf6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f7bf6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 1 -- 172.21.15.115:0/2601229236 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000e070 con 0x7f10300aecc0 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: dump: 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 1 -- 172.21.15.115:0/2601229236 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102000ba40 con 0x7f10300aecc0 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 1 -- 172.21.15.115:0/2601229236 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10200092e0 con 0x7f10300aecc0 2023-09-30T01:03:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f102dd9b700 10 monclient: dump: 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.601 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f1035199700 5 monclient: authenticate success, global_id 4292 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f1035199700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.519+0000 7f1035199700 4 set_mon_vals no callback set 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7fef7ad60700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7fef7ad60700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7fef7ad60700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef74072960 con 0x7fef7406e400 2023-09-30T01:03:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7f1035199700 10 monclient: shutdown 2023-09-30T01:03:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7f1035199700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7f1035199700 1 -- 172.21.15.115:0/2601229236 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 msgr2=0x7f10300b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.520+0000 7f1035199700 1 --2- 172.21.15.115:0/2601229236 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1020009f20 tx=0x7f1020009840 comp rx=0 tx=0).stop 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.530+0000 7f1035199700 1 -- 172.21.15.115:0/2601229236 shutdown_connections 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.530+0000 7f1035199700 1 --2- 172.21.15.115:0/2601229236 >> v2:172.21.15.115:3300/0 conn(0x7f10300ae8f0 0x7f10300a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.530+0000 7f1035199700 1 --2- 172.21.15.115:0/2601229236 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10300aecc0 0x7f10300b1470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.530+0000 7f1035199700 1 -- 172.21.15.115:0/2601229236 >> 172.21.15.115:0/2601229236 conn(0x7f103006a460 msgr2=0x7f10300630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.531+0000 7f1035199700 1 -- 172.21.15.115:0/2601229236 shutdown_connections 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.532+0000 7f1035199700 1 -- 172.21.15.115:0/2601229236 wait complete. 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.533+0000 7f1035199700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.533+0000 7f1035199700 10 monclient: monmap: 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 librados: starting msgr at 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 librados: starting objecter 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 Processor -- start 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 -- start start 2023-09-30T01:03:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 librados: setting wanted keys 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 1 librados: calling monclient init 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.534+0000 7f1035199700 10 monclient: init 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): picked mon.noname-a con 0x7f10300ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10300aecc0 0x7f103013e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): picked mon.noname-b con 0x7f10300aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10301415c0 0x7f10300a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): picked mon.noname-c con 0x7f10301415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103006d680 con 0x7f10300ae8f0 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10300a7740 con 0x7f10300aecc0 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103013eb40 con 0x7f10301415c0 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.535+0000 7f1035199700 10 monclient(hunting): authenticate will time out at 1079.985588s 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44324/0 (socket says 172.21.15.115:44324) 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 -- 172.21.15.115:0/1415753466 learned_addr learned my addr 172.21.15.115:0/1415753466 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): get_auth_request con 0x7f10300ae8f0 auth_method 0 2023-09-30T01:03:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef68052aa0 con 0x7fef7406e400 2023-09-30T01:03:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7fef70ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7fef70ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102e59c700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10300aecc0 0x7f103013e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102e59c700 10 monclient(hunting): get_auth_request con 0x7f10300aecc0 auth_method 0 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 -- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10301415c0 msgr2=0x7f10300a7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10301415c0 0x7f10300a7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 -- 172.21.15.115:0/1415753466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10300aecc0 msgr2=0x7f103013e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10300aecc0 0x7f103013e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 monclient: found mon.noname-a 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.536+0000 7f102ed9d700 1 -- 172.21.15.115:0/1415753466 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020009ae0 con 0x7f10300ae8f0 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f102ed9d700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f102400b600 tx=0x7f10240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 client.?.objecter ms_handle_connect 0x7f10300ae8f0 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400f510 con 0x7f10300ae8f0 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: dump: 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 5 monclient: authenticate success, global_id 4380 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 10 monclient: _renew_subs 2023-09-30T01:03:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103006bcd0 con 0x7f10300ae8f0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 10 monclient: _renew_subs 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103006de00 con 0x7f10300ae8f0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1035199700 1 librados: init done 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102400f880 con 0x7f10300ae8f0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400fa30 con 0x7f10300ae8f0 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f1016ffd700 10 monclient: dump: 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.537+0000 7f102d59a700 4 set_mon_vals no callback set 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.540+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f102400fb90 con 0x7f10300ae8f0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 1 --2- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 0x7f101c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f10240837a0 con 0x7f10300ae8f0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 10 client.4380.objecter ms_dispatch 0x7f1030001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 3 client.4380.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 3 client.4380.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.541+0000 7f1016ffd700 20 client.4380.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.544+0000 7f102e59c700 1 --2- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 0x7f101c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.544+0000 7f102e59c700 10 monclient: get_auth_request con 0x7f101c068910 auth_method 0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.544+0000 7f102e59c700 1 --2- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 0x7f101c06adc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ab0 tx=0x7f1020017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.546+0000 7f1035199700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.547+0000 7f1035199700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.547+0000 7f1035199700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.547+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10100050b0 con 0x7f10300ae8f0 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.549+0000 7f1016ffd700 10 client.4380.objecter ms_handle_connect 0x7f101c068910 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.555+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1024050e90 con 0x7f10300ae8f0 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.556+0000 7f1016ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.556+0000 7f1016ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.581+0000 7fcdfa60f700 10 monclient: get_monmap_and_config 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.581+0000 7fcdfa60f700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.581+0000 7fcdfa60f700 10 monclient: monmap: 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:38.582813+0000 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:38.582813+0000 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.619 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.583+0000 7fcdfa60f700 1 Processor -- start 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.583+0000 7fcdfa60f700 1 -- start start 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.583+0000 7fcdfa60f700 10 monclient: init 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T01:03:38.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdf406e400 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-b con 0x7fcdf406ec40 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdf413a8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf406bb30 con 0x7fcdf413a8f0 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf413dde0 con 0x7fcdf406e400 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf406d840 con 0x7fcdf406ec40 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.584+0000 7fcdfa60f700 10 monclient(hunting): authenticate will time out at 1080.034464s 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59272/0 (socket says 172.21.15.115:59272) 2023-09-30T01:03:38.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 -- 172.21.15.115:0/1728122424 learned_addr learned my addr 172.21.15.115:0/1728122424 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): get_auth_request con 0x7fcdf406ec40 auth_method 0 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 -- 172.21.15.115:0/1728122424 >> v1:172.21.15.157:6789/0 conn(0x7fcdf406e400 legacy=0x7fcdf413d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 -- 172.21.15.115:0/1728122424 >> v1:172.21.15.115:6789/0 conn(0x7fcdf413a8f0 legacy=0x7fcdf4144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 monclient: found mon.noname-b 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.585+0000 7fcdf37fe700 1 -- 172.21.15.115:0/1728122424 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40aab40 con 0x7fcdf406ec40 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.586+0000 7fcdf37fe700 1 --2- 172.21.15.115:0/1728122424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d050 tx=0x7fcde40082d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1472afff1936b356 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 1 -- 172.21.15.115:0/1728122424 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4010070 con 0x7fcdf406ec40 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: dump: 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.623 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 1 -- 172.21.15.115:0/1728122424 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde400dae0 con 0x7fcdf406ec40 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 1 -- 172.21.15.115:0/1728122424 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde400de80 con 0x7fcdf406ec40 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdf2ffd700 10 monclient: dump: 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.625 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdfa60f700 5 monclient: authenticate success, global_id 4295 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdfa60f700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.589+0000 7fcdfa60f700 4 set_mon_vals no callback set 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.590+0000 7fcdfa60f700 10 monclient: shutdown 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.590+0000 7fcdfa60f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.590+0000 7fcdfa60f700 1 -- 172.21.15.115:0/1728122424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 msgr2=0x7fcdf413a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.590+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/1728122424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d050 tx=0x7fcde40082d0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.591+0000 7fcdfa60f700 1 -- 172.21.15.115:0/1728122424 shutdown_connections 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.591+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/1728122424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf406ec40 0x7fcdf413a3b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.591+0000 7fcdfa60f700 1 -- 172.21.15.115:0/1728122424 >> 172.21.15.115:0/1728122424 conn(0x7fcdf4067bd0 msgr2=0x7fcdf40b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.591+0000 7fcdfa60f700 1 -- 172.21.15.115:0/1728122424 shutdown_connections 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.591+0000 7fcdfa60f700 1 -- 172.21.15.115:0/1728122424 wait complete. 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.592+0000 7fcdfa60f700 10 monclient: build_initial_monmap 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.592+0000 7fcdfa60f700 10 monclient: monmap: 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:38.627 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 librados: starting msgr at 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 librados: starting objecter 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 Processor -- start 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 -- start start 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 librados: setting wanted keys 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 1 librados: calling monclient init 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.593+0000 7fcdfa60f700 10 monclient: init 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:38.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-a con 0x7fcdf406e400 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 0x7fcdf40a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-b con 0x7fcdf406ec40 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf413a8f0 0x7fcdf413b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): picked mon.noname-c con 0x7fcdf413a8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf413aed0 con 0x7fcdf406e400 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40a7b20 con 0x7fcdf406ec40 2023-09-30T01:03:38.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf413c8f0 con 0x7fcdf413a8f0 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.594+0000 7fcdfa60f700 10 monclient(hunting): authenticate will time out at 1080.044813s 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44334/0 (socket says 172.21.15.115:44334) 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 1 -- 172.21.15.115:0/3676848186 learned_addr learned my addr 172.21.15.115:0/3676848186 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): get_auth_request con 0x7fcdf406e400 auth_method 0 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 0x7fcdf40a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): get_auth_request con 0x7fcdf406ec40 auth_method 0 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 -- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf413a8f0 msgr2=0x7fcdf413b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf413a8f0 0x7fcdf413b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 -- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 msgr2=0x7fcdf40a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 monclient: found mon.noname-b 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.595+0000 7fcdf37fe700 1 -- 172.21.15.115:0/3676848186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4008570 con 0x7fcdf406ec40 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.596+0000 7fcdf37fe700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 0x7fcdf40a6460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406f710 tx=0x7fcde401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.597+0000 7fcdf3fff700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 client.?.objecter ms_handle_connect 0x7fcdf406ec40 2023-09-30T01:03:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4016c20 con 0x7fcdf406ec40 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: dump: 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde4016d80 con 0x7fcdf406ec40 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde400b310 con 0x7fcdf406ec40 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcddbfff700 10 monclient: dump: 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:38.634 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 5 monclient: authenticate success, global_id 4387 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 10 monclient: _renew_subs 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf413b8a0 con 0x7fcdf406ec40 2023-09-30T01:03:38.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 10 monclient: _renew_subs 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf413bd90 con 0x7fcdf406ec40 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdfa60f700 1 librados: init done 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.598+0000 7fcdf27fc700 4 set_mon_vals no callback set 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.602+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcde401d040 con 0x7fcdf406ec40 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.602+0000 7fcddbfff700 1 --2- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 0x7fcde006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcde4084bd0 con 0x7fcdf406ec40 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcddbfff700 10 client.4387.objecter ms_dispatch 0x7fcdf4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcddbfff700 3 client.4387.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcddbfff700 3 client.4387.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:38.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcddbfff700 20 client.4387.objecter dump_active .. 0 homeless 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdfa60f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdfa60f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdfa60f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd40050b0 con 0x7fcdf406ec40 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdf3fff700 1 --2- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 0x7fcde006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.603+0000 7fcdf3fff700 10 monclient: get_auth_request con 0x7fcde00689e0 auth_method 0 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.604+0000 7fcdf3fff700 1 --2- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 0x7fcde006ae90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009f60 tx=0x7fcde80091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.604+0000 7fcddbfff700 10 client.4387.objecter ms_handle_connect 0x7fcde00689e0 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.608+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcde4052470 con 0x7fcdf406ec40 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.608+0000 7fcddbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:38.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.608+0000 7fcddbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fceedec4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fceedec4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fceedec4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcee8072960 con 0x7fcee80a61a0 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fcecf7fe700 1 -- 172.21.15.115:0/1081586874 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcee80a9180 con 0x7fcee80a61a0 2023-09-30T01:03:38.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fcecf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T01:03:38.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.831+0000 7fcecf7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.835 INFO:teuthology.orchestra.run.smithi115.stdout:55834574852 2023-09-30T01:03:38.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 10 librados: watch_flush enter 2023-09-30T01:03:38.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 10 librados: watch_flush exit 2023-09-30T01:03:38.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 20 client.4277.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 10 client.4277.objecter successfully canceled tick 2023-09-30T01:03:38.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 msgr2=0x7fced406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 0x7fced406ae90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fced8006e80 tx=0x7fced8016000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 10 monclient: shutdown 2023-09-30T01:03:38.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 msgr2=0x7fcee80a6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.835+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 0x7fcee80a6620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcee000bb00 tx=0x7fcee0009780 comp rx=0 tx=0).stop 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 shutdown_connections 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> v2:172.21.15.115:6816/115635 conn(0x7fced40689e0 0x7fced406ae90 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fced8006e80 tx=0x7fced8016000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcee80a61a0 0x7fcee80a6620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcee806b800 0x7fcee80a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 --2- 172.21.15.115:0/1081586874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcee806e1f0 0x7fcee80a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 >> 172.21.15.115:0/1081586874 conn(0x7fcee80b0540 msgr2=0x7fcee806ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 shutdown_connections 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 -- 172.21.15.115:0/1081586874 wait complete. 2023-09-30T01:03:38.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.836+0000 7fceedec4700 1 librados: shutdown 2023-09-30T01:03:38.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.848+0000 7f2bb885e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:38.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.848+0000 7f2bb885e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:38.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.848+0000 7f2bb885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.848+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2bb406dd60 con 0x7f2bb40aecc0 2023-09-30T01:03:38.852 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:38.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.849+0000 7f2ba27fc700 1 -- 172.21.15.115:0/3852377059 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2bb406dd60 con 0x7f2bb40aecc0 2023-09-30T01:03:38.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.849+0000 7f2ba27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:03:38.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.849+0000 7f2ba27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 10 librados: watch_flush enter 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 10 librados: watch_flush exit 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 20 client.4359.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 10 client.4359.objecter successfully canceled tick 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 msgr2=0x7f2b9406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 0x7f2b9406ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8006d30 comp rx=0 tx=0).stop 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 10 monclient: shutdown 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 msgr2=0x7f2bb406e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.852+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40090b0 tx=0x7f2ba4017000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2023-09-30T01:03:38.855 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 shutdown_connections 2023-09-30T01:03:38.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> v2:172.21.15.115:6816/115635 conn(0x7f2b94068960 0x7f2b9406ae10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2bb41415c0 0x7f2bb406c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2bb40ae8f0 0x7f2bb406be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 --2- 172.21.15.115:0/3852377059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bb40aecc0 0x7f2bb406e5e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 >> 172.21.15.115:0/3852377059 conn(0x7f2bb406a460 msgr2=0x7f2bb413fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 shutdown_connections 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 -- 172.21.15.115:0/3852377059 wait complete. 2023-09-30T01:03:38.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.853+0000 7f2bb885e700 1 librados: shutdown 2023-09-30T01:03:38.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2023-09-30T01:03:38.866 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f3d883700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f3d883700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f3d883700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9f3806b210 con 0x7f9f380b0c50 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f1effd700 1 -- 172.21.15.115:0/3132610591 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9f300576c0 con 0x7f9f380b0c50 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.889+0000 7f9f1effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.890 INFO:teuthology.orchestra.run.smithi115.stdout:55834574852 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.890+0000 7f25406d8700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.890+0000 7f25406d8700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.890+0000 7f25406d8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.890+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2538152900 con 0x7f25380b0c50 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.891+0000 7f252e7fc700 1 -- 172.21.15.115:0/133642469 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2534018090 con 0x7f25380b0c50 2023-09-30T01:03:38.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.891+0000 7f252e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:03:38.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.891+0000 7f252e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.892 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 10 librados: watch_flush enter 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 10 librados: watch_flush exit 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 20 client.4371.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 10 client.4371.objecter successfully canceled tick 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 msgr2=0x7f9f2406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 0x7f9f2406ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c006d40 tx=0x7f9f2c008040 comp rx=0 tx=0).stop 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 10 monclient: shutdown 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 msgr2=0x7f9f380a8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.892+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f380b13d0 tx=0x7f9f30019000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 shutdown_connections 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> v2:172.21.15.115:6816/115635 conn(0x7f9f24068890 0x7f9f2406ad40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f380b0c50 0x7f9f380a8260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f3806e140 0x7f9f3812ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 --2- 172.21.15.115:0/3132610591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f9f3806e5a0 0x7f9f3812ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.893+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 >> 172.21.15.115:0/3132610591 conn(0x7f9f38067bd0 msgr2=0x7f9f380ae590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 shutdown_connections 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 10 librados: watch_flush enter 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 10 librados: watch_flush exit 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 20 client.4381.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 10 client.4381.objecter successfully canceled tick 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 msgr2=0x7f251c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 0x7f251c06ae10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2528006fd0 tx=0x7f2528006db0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 10 monclient: shutdown 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 msgr2=0x7f25381467e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f253400cb90 tx=0x7f2534001a80 comp rx=0 tx=0).stop 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 shutdown_connections 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> v2:172.21.15.115:6816/115635 conn(0x7f251c068960 0x7f251c06ae10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f253813a3b0 0x7f2538154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f253806e400 0x7f2538151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 --2- 172.21.15.115:0/133642469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25380b0c50 0x7f25381467e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 >> 172.21.15.115:0/133642469 conn(0x7f2538067bd0 msgr2=0x7f25380acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 shutdown_connections 2023-09-30T01:03:38.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.894+0000 7f25406d8700 1 -- 172.21.15.115:0/133642469 wait complete. 2023-09-30T01:03:38.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.895+0000 7f9f3d883700 1 -- 172.21.15.115:0/3132610591 wait complete. 2023-09-30T01:03:38.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.895+0000 7f25406d8700 1 librados: shutdown 2023-09-30T01:03:38.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.895+0000 7f9f3d883700 1 librados: shutdown 2023-09-30T01:03:38.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7c1548a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:38.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7c1548a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:38.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7c1548a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7c1006e7d0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7bf6ffd700 1 -- 172.21.15.115:0/1353246431 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c000548e0 con 0x7f7c100ae8f0 2023-09-30T01:03:38.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7bf6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:03:38.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.908+0000 7f7bf6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.909 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T01:03:38.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.911+0000 7f7c1548a700 10 librados: watch_flush enter 2023-09-30T01:03:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2023-09-30T01:03:38.913 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.911+0000 7f7c1548a700 10 librados: watch_flush exit 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 20 client.4374.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 10 client.4374.objecter successfully canceled tick 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 msgr2=0x7f7bfc06ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 0x7f7bfc06ae40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c040058d0 tx=0x7f7c04007ac0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 10 monclient: shutdown 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 msgr2=0x7f7c100a8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000b5e0 tx=0x7f7c000090f0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 shutdown_connections 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> v2:172.21.15.115:6816/115635 conn(0x7f7bfc068990 0x7f7bfc06ae40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7c100aecc0 0x7f7c10152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c100a5d10 0x7f7c100a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 --2- 172.21.15.115:0/1353246431 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7c100ae8f0 0x7f7c100a8df0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 >> 172.21.15.115:0/1353246431 conn(0x7f7c1006a460 msgr2=0x7f7c100b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 shutdown_connections 2023-09-30T01:03:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 -- 172.21.15.115:0/1353246431 wait complete. 2023-09-30T01:03:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.912+0000 7f7c1548a700 1 librados: shutdown 2023-09-30T01:03:38.916 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-30T01:03:38.916 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef7ad60700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:38.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef7ad60700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef7ad60700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fef7406e7d0 con 0x7fef7406e400 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef70ff9700 1 -- 172.21.15.115:0/1953134208 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef68026070 con 0x7fef7406e400 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef70ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.922+0000 7fef70ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.923 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:38.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 10 librados: watch_flush enter 2023-09-30T01:03:38.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 10 librados: watch_flush exit 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 20 client.4377.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 10 client.4377.objecter successfully canceled tick 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 msgr2=0x7fef6006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 0x7fef6006ae10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef7413d1a0 tx=0x7fef6c0016e0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 10 monclient: shutdown 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 msgr2=0x7fef740a7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef740aac80 tx=0x7fef6801c000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 shutdown_connections 2023-09-30T01:03:38.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> v2:172.21.15.115:6816/115635 conn(0x7fef60068960 0x7fef6006ae10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef7413db30 0x7fef74135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef7406e400 0x7fef740a7020 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 --2- 172.21.15.115:0/1953134208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef7413d6d0 0x7fef74134ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 >> 172.21.15.115:0/1953134208 conn(0x7fef74067bd0 msgr2=0x7fef740acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 shutdown_connections 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 -- 172.21.15.115:0/1953134208 wait complete. 2023-09-30T01:03:38.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.925+0000 7fef7ad60700 1 librados: shutdown 2023-09-30T01:03:38.929 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.1 2023-09-30T01:03:38.929 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1035199700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1035199700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1035199700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1010005360 con 0x7f10300ae8f0 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1016ffd700 1 -- 172.21.15.115:0/1415753466 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1024054680 con 0x7f10300ae8f0 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1016ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.934+0000 7f1016ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.935 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 10 librados: watch_flush enter 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 10 librados: watch_flush exit 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 20 client.4380.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 10 client.4380.objecter successfully canceled tick 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 msgr2=0x7f101c06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 0x7f101c06adc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ab0 tx=0x7f1020017000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 10 monclient: shutdown 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 msgr2=0x7f103006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.936+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f102400b600 tx=0x7f10240091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 shutdown_connections 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> v2:172.21.15.115:6816/115635 conn(0x7f101c068910 0x7f101c06adc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10301415c0 0x7f10300a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10300ae8f0 0x7f103006c910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 --2- 172.21.15.115:0/1415753466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10300aecc0 0x7f103013e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 >> 172.21.15.115:0/1415753466 conn(0x7f103006a460 msgr2=0x7f103013f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 shutdown_connections 2023-09-30T01:03:38.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 -- 172.21.15.115:0/1415753466 wait complete. 2023-09-30T01:03:38.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.937+0000 7f1035199700 1 librados: shutdown 2023-09-30T01:03:38.940 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.3 2023-09-30T01:03:38.941 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.2 2023-09-30T01:03:38.948 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.960+0000 7fcdfa60f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.960+0000 7fcdfa60f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.960+0000 7fcdfa60f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.157:3300/0 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.960+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcdd4005360 con 0x7fcdf406ec40 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.961+0000 7fcddbfff700 1 -- 172.21.15.115:0/3676848186 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcde4026020 con 0x7fcdf406ec40 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.961+0000 7fcddbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:03:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.961+0000 7fcddbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:38.962 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T01:03:38.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 10 librados: watch_flush enter 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 10 librados: watch_flush exit 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 20 client.4387.objecter shutdown clearing up homeless session... 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 10 client.4387.objecter successfully canceled tick 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 msgr2=0x7fcde006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 0x7fcde006ae90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009f60 tx=0x7fcde80091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 10 monclient: shutdown 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 msgr2=0x7fcdf40a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 0x7fcdf40a6460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406f710 tx=0x7fcde401b000 comp rx=0 tx=0).stop 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 shutdown_connections 2023-09-30T01:03:38.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> v2:172.21.15.115:6816/115635 conn(0x7fcde00689e0 0x7fcde006ae90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdf413a8f0 0x7fcdf413b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdf406e400 0x7fcdf40a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 --2- 172.21.15.115:0/3676848186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdf406ec40 0x7fcdf40a6460 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 >> 172.21.15.115:0/3676848186 conn(0x7fcdf4067bd0 msgr2=0x7fcdf406b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 shutdown_connections 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 -- 172.21.15.115:0/3676848186 wait complete. 2023-09-30T01:03:38.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:38.963+0000 7fcdfa60f700 1 librados: shutdown 2023-09-30T01:03:38.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.0 2023-09-30T01:03:38.975 DEBUG:teuthology.parallel:result is None 2023-09-30T01:03:38.975 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:03:38.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:03:39.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 10 monclient: get_monmap_and_config 2023-09-30T01:03:39.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 10 monclient: monmap: 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:39.133210+0000 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:39.133210+0000 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:39.133 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:39.134 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:39.134 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:39.134 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:39.134 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 1 Processor -- start 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 1 -- start start 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.132+0000 7f696d435700 10 monclient: init 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-09-30T01:03:39.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69680ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f69680aecc0 0x7f69680a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): picked mon.noname-c con 0x7f69680aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f69681415c0 0x7f69680adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): picked mon.noname-a con 0x7f69681415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69680a7040 con 0x7f69680ae8f0 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69680a57f0 con 0x7f69681415c0 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69680a5480 con 0x7f69680aecc0 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f696d435700 10 monclient(hunting): authenticate will time out at 1080.583447s 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f69681415c0 0x7f69680adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f69681415c0 0x7f69680adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44344/0 (socket says 172.21.15.115:44344) 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 -- 172.21.15.115:0/1936101685 learned_addr learned my addr 172.21.15.115:0/1936101685 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f6966ffd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f69680ae8f0 0x7f69680b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46718/0 (socket says 172.21.15.115:46718) 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): get_auth_request con 0x7f69681415c0 auth_method 0 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69667fc700 1 --2- 172.21.15.115:0/1936101685 >> v2:172.21.15.157:3300/0 conn(0x7f69680aecc0 0x7f69680a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69667fc700 10 monclient(hunting): get_auth_request con 0x7f69680aecc0 auth_method 0 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 -- 172.21.15.115:0/1936101685 >> v2:172.21.15.157:3300/0 conn(0x7f69680aecc0 msgr2=0x7f69680a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:39.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 --2- 172.21.15.115:0/1936101685 >> v2:172.21.15.157:3300/0 conn(0x7f69680aecc0 0x7f69680a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 -- 172.21.15.115:0/1936101685 >> v1:172.21.15.115:6789/0 conn(0x7f69680ae8f0 legacy=0x7f69680b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f69677fe700 1 -- 172.21.15.115:0/1936101685 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680a55c0 con 0x7f69681415c0 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f6965ffb700 1 -- 172.21.15.115:0/1936101685 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760272642 0 0) 0x7f69680a7040 con 0x7f69680ae8f0 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.133+0000 7f6965ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f69677fe700 1 --2- 172.21.15.115:0/1936101685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680adb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6968064080 tx=0x7f6960009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e45620ee153493 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 1 -- 172.21.15.115:0/1936101685 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000e070 con 0x7f69681415c0 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: dump: 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.147 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 1 -- 172.21.15.115:0/1936101685 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000ba50 con 0x7f69681415c0 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 5 monclient: authenticate success, global_id 4386 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 1 -- 172.21.15.115:0/1936101685 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600092e0 con 0x7f69681415c0 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:39.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 4 set_mon_vals no callback set 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f6965ffb700 10 monclient: dump: 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.149 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 10 monclient: shutdown 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 -- 172.21.15.115:0/1936101685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 msgr2=0x7f69680adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 --2- 172.21.15.115:0/1936101685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680adb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6968064080 tx=0x7f6960009840 comp rx=0 tx=0).stop 2023-09-30T01:03:39.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 -- 172.21.15.115:0/1936101685 shutdown_connections 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 --2- 172.21.15.115:0/1936101685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680adb60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 --2- 172.21.15.115:0/1936101685 >> v2:172.21.15.157:3300/0 conn(0x7f69680aecc0 0x7f69680a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 -- 172.21.15.115:0/1936101685 >> 172.21.15.115:0/1936101685 conn(0x7f696806a460 msgr2=0x7f69680630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 -- 172.21.15.115:0/1936101685 shutdown_connections 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.134+0000 7f696d435700 1 -- 172.21.15.115:0/1936101685 wait complete. 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.135+0000 7f696d435700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.135+0000 7f696d435700 10 monclient: monmap: 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:39.151 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.135+0000 7f696d435700 1 librados: starting msgr at 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.135+0000 7f696d435700 1 librados: starting objecter 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.135+0000 7f696d435700 1 Processor -- start 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 -- start start 2023-09-30T01:03:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 librados: setting wanted keys 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 librados: calling monclient init 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient: init 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): picked mon.noname-c con 0x7f69680ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69680aecc0 0x7f696814e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): picked mon.noname-b con 0x7f69680aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:39.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): picked mon.noname-a con 0x7f69681415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696814daa0 con 0x7f69681415c0 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696814dd80 con 0x7f69680aecc0 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69680aaeb0 con 0x7f69680ae8f0 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f696d435700 10 monclient(hunting): authenticate will time out at 1080.586630s 2023-09-30T01:03:39.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59282/0 (socket says 172.21.15.115:59282) 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44346/0 (socket says 172.21.15.115:44346) 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 1 -- 172.21.15.115:0/1249644489 learned_addr learned my addr 172.21.15.115:0/1249644489 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): get_auth_request con 0x7f69680ae8f0 auth_method 0 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): get_auth_request con 0x7f69681415c0 auth_method 0 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69680aecc0 0x7f696814e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f6966ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): get_auth_request con 0x7f69680aecc0 auth_method 0 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.136+0000 7f69667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 -- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 msgr2=0x7f696813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 -- 172.21.15.115:0/1249644489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69680aecc0 msgr2=0x7f696814e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69680aecc0 0x7f696814e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:39.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 -- 172.21.15.115:0/1249644489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960009ae0 con 0x7f69681415c0 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f6966ffd700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69677fe700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69680b59c0 tx=0x7f6960018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 client.?.objecter ms_handle_connect 0x7f69681415c0 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000be60 con 0x7f69681415c0 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: dump: 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.158 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 5 monclient: authenticate success, global_id 4392 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 10 monclient: _renew_subs 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960008d20 con 0x7f69681415c0 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69680a7ba0 con 0x7f69681415c0 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696001c680 con 0x7f69681415c0 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f694effd700 10 monclient: dump: 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.160 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 10 monclient: _renew_subs 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69680a9b40 con 0x7f69681415c0 2023-09-30T01:03:39.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f696d435700 1 librados: init done 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.137+0000 7f69657fa700 4 set_mon_vals no callback set 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.138+0000 7f696d435700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.138+0000 7f696d435700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.138+0000 7f696d435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.138+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69680aad20 con 0x7f69681415c0 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6960003040 con 0x7f69681415c0 2023-09-30T01:03:39.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 1 --2- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 0x7f695406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f6966ffd700 1 --2- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 0x7f695406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6960085c70 con 0x7f69681415c0 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 10 client.4392.objecter ms_dispatch 0x7f6968001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 3 client.4392.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 3 client.4392.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f6966ffd700 10 monclient: get_auth_request con 0x7f6954068890 auth_method 0 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 20 client.4392.objecter dump_active .. 0 homeless 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f694effd700 10 client.4392.objecter ms_handle_connect 0x7f6954068890 2023-09-30T01:03:39.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.139+0000 7f6966ffd700 1 --2- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 0x7f695406ad40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f695c007fd0 tx=0x7f695c007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.142+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69600535b0 con 0x7f69681415c0 2023-09-30T01:03:39.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.143+0000 7f694effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.143+0000 7f694effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:39.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.321+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69680a9df0 con 0x7f6954068890 2023-09-30T01:03:39.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.323+0000 7f694effd700 1 -- 172.21.15.115:0/1249644489 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47998 (secure 0 0 0) 0x7f69680a9df0 con 0x7f6954068890 2023-09-30T01:03:39.324 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:39.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 10 librados: watch_flush enter 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 10 librados: watch_flush exit 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 20 client.4392.objecter shutdown clearing up homeless session... 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 10 client.4392.objecter successfully canceled tick 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 msgr2=0x7f695406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 0x7f695406ad40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f695c007fd0 tx=0x7f695c007e10 comp rx=0 tx=0).stop 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 10 monclient: shutdown 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 msgr2=0x7f69680a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69680b59c0 tx=0x7f6960018000 comp rx=0 tx=0).stop 2023-09-30T01:03:39.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 shutdown_connections 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> v2:172.21.15.115:6816/115635 conn(0x7f6954068890 0x7f695406ad40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f69681415c0 0x7f69680a9600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f69680ae8f0 0x7f696813d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 --2- 172.21.15.115:0/1249644489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69680aecc0 0x7f696814e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 >> 172.21.15.115:0/1249644489 conn(0x7f696806a460 msgr2=0x7f69680b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 shutdown_connections 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 -- 172.21.15.115:0/1249644489 wait complete. 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.325+0000 7f696d435700 1 librados: shutdown 2023-09-30T01:03:39.327 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:03:39.337 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T01:03:38.002063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41892,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932828,"statfs":{"total":773094113280,"available":773051215872,"internally_reserved":0,"allocated":1253376,"data_stored":1407216,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033168+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033168+0000","last_peered":"2023-09-30T01:03:35.033168+0000","last_clean":"2023-09-30T01:03:35.033168+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033168+0000","last_undegraded":"2023-09-30T01:03:35.033168+0000","last_fullsized":"2023-09-30T01:03:35.033168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]}]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2469999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.925}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.649}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.895}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5636,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366204,"statfs":{"total":96636764160,"available":96630992896,"internally_reserved":0,"allocated":192512,"data_stored":570290,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:03:39.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 10 monclient: get_monmap_and_config 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 10 monclient: monmap: 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:39.456106+0000 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:39.456106+0000 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:39.456 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 1 Processor -- start 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 1 -- start start 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.455+0000 7f1f6b1e1700 10 monclient: init 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1f640ae8f0 0x7f1f640a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-c con 0x7f1f640ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f640aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:39.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f641415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640a7040 con 0x7f1f641415c0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640a57f0 con 0x7f1f640aecc0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f640a5480 con 0x7f1f640ae8f0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6b1e1700 10 monclient(hunting): authenticate will time out at 1080.906592s 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6977e700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f1f641415c0 0x7f1f640adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:46730/0 (socket says 172.21.15.115:46730) 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f6977e700 1 -- 172.21.15.115:0/2522387926 learned_addr learned my addr 172.21.15.115:0/2522387926 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 1 --2- 172.21.15.115:0/2522387926 >> v2:172.21.15.157:3300/0 conn(0x7f1f640ae8f0 0x7f1f640a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): get_auth_request con 0x7f1f640ae8f0 auth_method 0 2023-09-30T01:03:39.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907255311 0 0) 0x7f1f640a7040 con 0x7f1f641415c0 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f637fe700 10 monclient(hunting): my global_id is 4395 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f54003570 con 0x7f1f641415c0 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.456+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988972861 0 0) 0x7f1f640a57f0 con 0x7f1f640aecc0 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient(hunting): my global_id is 4399 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f640a7040 con 0x7f1f640aecc0 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 1 -- 172.21.15.115:0/2522387926 >> v1:172.21.15.157:6789/0 conn(0x7f1f640aecc0 legacy=0x7f1f640b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021400548 0 0) 0x7f1f54003570 con 0x7f1f641415c0 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 1 -- 172.21.15.115:0/2522387926 >> v1:172.21.15.115:6789/0 conn(0x7f1f641415c0 legacy=0x7f1f640adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 1 monclient: found mon.noname-c 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 1 -- 172.21.15.115:0/2522387926 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640a55c0 con 0x7f1f640ae8f0 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f68f7d700 1 --2- 172.21.15.115:0/2522387926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f640a5d10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f580098e0 tx=0x7f1f580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=976923469173acbb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0017f0 con 0x7f1f640ae8f0 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: dump: 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.469 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f6b1e1700 5 monclient: authenticate success, global_id 4396 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f6b1e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58011420 con 0x7f1f640ae8f0 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f6b1e1700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 1 -- 172.21.15.115:0/2522387926 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f580117c0 con 0x7f1f640ae8f0 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f6b1e1700 4 set_mon_vals no callback set 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.457+0000 7f1f637fe700 10 monclient: dump: 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.471 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 10 monclient: shutdown 2023-09-30T01:03:39.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/2522387926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 msgr2=0x7f1f640a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/2522387926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f640a5d10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f580098e0 tx=0x7f1f580091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/2522387926 shutdown_connections 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/2522387926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f640a5d10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/2522387926 >> 172.21.15.115:0/2522387926 conn(0x7f1f6406a460 msgr2=0x7f1f640630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/2522387926 shutdown_connections 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/2522387926 wait complete. 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 10 monclient: monmap: 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:39.473 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 librados: starting msgr at 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.458+0000 7f1f6b1e1700 1 librados: starting objecter 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 Processor -- start 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 -- start start 2023-09-30T01:03:39.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 librados: setting wanted keys 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 librados: calling monclient init 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient: init 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f6413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-b con 0x7f1f640ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 0x7f1f6414dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-c con 0x7f1f640aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): picked mon.noname-a con 0x7f1f641415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640aac20 con 0x7f1f641415c0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640a89c0 con 0x7f1f640ae8f0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640a9600 con 0x7f1f640aecc0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6b1e1700 10 monclient(hunting): authenticate will time out at 1080.909736s 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 0x7f1f6414dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44352/0 (socket says 172.21.15.115:44352) 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 1 -- 172.21.15.115:0/3046268725 learned_addr learned my addr 172.21.15.115:0/3046268725 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): get_auth_request con 0x7f1f641415c0 auth_method 0 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): get_auth_request con 0x7f1f640aecc0 auth_method 0 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f6977e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.459+0000 7f1f68f7d700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f6413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): get_auth_request con 0x7f1f640ae8f0 auth_method 0 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f68f7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 -- 172.21.15.115:0/3046268725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 msgr2=0x7f1f6413e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f6413e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 -- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 msgr2=0x7f1f6414e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 monclient: found mon.noname-c 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 -- 172.21.15.115:0/3046268725 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58009420 con 0x7f1f640aecc0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6977e700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f63fff700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 0x7f1f6414dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c560 tx=0x7f1f50007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 client.?.objecter ms_handle_connect 0x7f1f640aecc0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50011590 con 0x7f1f640aecc0 2023-09-30T01:03:39.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: dump: 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.480 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f500116f0 con 0x7f1f640aecc0 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 5 monclient: authenticate success, global_id 4301 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 10 monclient: _renew_subs 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6414e6c0 con 0x7f1f640aecc0 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50011ab0 con 0x7f1f640aecc0 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f60ff9700 10 monclient: dump: 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.482 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 10 monclient: _renew_subs 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f62ffd700 4 set_mon_vals no callback set 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f640aa280 con 0x7f1f640aecc0 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.460+0000 7f1f6b1e1700 1 librados: init done 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.461+0000 7f1f6b1e1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.461+0000 7f1f6b1e1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.461+0000 7f1f6b1e1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.461+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f640aa7e0 con 0x7f1f640aecc0 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.461+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1f5001c460 con 0x7f1f640aecc0 2023-09-30T01:03:39.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 1 --2- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 0x7f1f4806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1f50083aa0 con 0x7f1f640aecc0 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 10 client.4301.objecter ms_dispatch 0x7f1f64001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 3 client.4301.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 3 client.4301.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f68f7d700 1 --2- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 0x7f1f4806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f68f7d700 10 monclient: get_auth_request con 0x7f1f48068890 auth_method 0 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 20 client.4301.objecter dump_active .. 0 homeless 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f60ff9700 10 client.4301.objecter ms_handle_connect 0x7f1f48068890 2023-09-30T01:03:39.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.462+0000 7f1f68f7d700 1 --2- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 0x7f1f4806ad40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f580093f0 tx=0x7f1f58016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.466+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f640aa7e0 con 0x7f1f640aecc0 2023-09-30T01:03:39.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.466+0000 7f1f60ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.466+0000 7f1f60ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:39.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.645+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f640aa7e0 con 0x7f1f48068890 2023-09-30T01:03:39.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.647+0000 7f1f60ff9700 1 -- 172.21.15.115:0/3046268725 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47998 (secure 0 0 0) 0x7f1f640aa7e0 con 0x7f1f48068890 2023-09-30T01:03:39.647 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 10 librados: watch_flush enter 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 10 librados: watch_flush exit 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 20 client.4301.objecter shutdown clearing up homeless session... 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 10 client.4301.objecter successfully canceled tick 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 msgr2=0x7f1f4806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 0x7f1f4806ad40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f580093f0 tx=0x7f1f58016000 comp rx=0 tx=0).stop 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 10 monclient: shutdown 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 msgr2=0x7f1f6414dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 0x7f1f6414dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c560 tx=0x7f1f50007d50 comp rx=0 tx=0).stop 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 shutdown_connections 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> v2:172.21.15.115:6816/115635 conn(0x7f1f48068890 0x7f1f4806ad40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f641415c0 0x7f1f6414e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f640ae8f0 0x7f1f6413e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 --2- 172.21.15.115:0/3046268725 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1f640aecc0 0x7f1f6414dc40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 >> 172.21.15.115:0/3046268725 conn(0x7f1f6406a460 msgr2=0x7f1f6413abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 shutdown_connections 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 -- 172.21.15.115:0/3046268725 wait complete. 2023-09-30T01:03:39.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.649+0000 7f1f6b1e1700 1 librados: shutdown 2023-09-30T01:03:39.652 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:03:39.661 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T01:03:38.002063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41892,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932828,"statfs":{"total":773094113280,"available":773051215872,"internally_reserved":0,"allocated":1253376,"data_stored":1407216,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033168+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033168+0000","last_peered":"2023-09-30T01:03:35.033168+0000","last_clean":"2023-09-30T01:03:35.033168+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033168+0000","last_undegraded":"2023-09-30T01:03:35.033168+0000","last_fullsized":"2023-09-30T01:03:35.033168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]}]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":41362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2469999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.925}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.649}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.895}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5636,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366204,"statfs":{"total":96636764160,"available":96630992896,"internally_reserved":0,"allocated":192512,"data_stored":570290,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:03:39.662 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:03:39.662 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T01:03:39.663 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T01:03:39.663 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:39.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.778+0000 7fc31ba09700 10 monclient: get_monmap_and_config 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.778+0000 7fc31ba09700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.778+0000 7fc31ba09700 10 monclient: monmap: 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:39.779527+0000 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:39.779527+0000 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:39.779 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:39.780 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:39.780 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 Processor -- start 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 -- start start 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient: init 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc3140ae8f0 0x7fc3140a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-a con 0x7fc3140ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 0x7fc3140b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-c con 0x7fc3140aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3141415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3140a7040 con 0x7fc3141415c0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140b19b0 con 0x7fc3140ae8f0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140a57f0 con 0x7fc3140aecc0 2023-09-30T01:03:39.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.779+0000 7fc31ba09700 10 monclient(hunting): authenticate will time out at 1081.229921s 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc3140ae8f0 0x7fc3140a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc3140ae8f0 0x7fc3140a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44368/0 (socket says 172.21.15.115:44368) 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 -- 172.21.15.115:0/3208236068 learned_addr learned my addr 172.21.15.115:0/3208236068 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): get_auth_request con 0x7fc3140ae8f0 auth_method 0 2023-09-30T01:03:39.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 1 --2- 172.21.15.115:0/3208236068 >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 0x7fc3140b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 10 monclient(hunting): get_auth_request con 0x7fc3140aecc0 auth_method 0 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2023-09-30T01:03:39.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 -- 172.21.15.115:0/3208236068 >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 msgr2=0x7fc3140b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 --2- 172.21.15.115:0/3208236068 >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 0x7fc3140b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 -- 172.21.15.115:0/3208236068 >> v1:172.21.15.157:6789/0 conn(0x7fc3141415c0 legacy=0x7fc3140adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 monclient: found mon.noname-a 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 -- 172.21.15.115:0/3208236068 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140a5480 con 0x7fc3140ae8f0 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc318fa4700 1 --2- 172.21.15.115:0/3208236068 >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 0x7fc3140b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc3197a5700 1 --2- 172.21.15.115:0/3208236068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140ae8f0 0x7fc3140a5d10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc314064080 tx=0x7fc310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfda0d1629bd700d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 1 -- 172.21.15.115:0/3208236068 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000b860 con 0x7fc3140ae8f0 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: dump: 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc31ba09700 5 monclient: authenticate success, global_id 4401 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc31ba09700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 1 -- 172.21.15.115:0/3208236068 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31000b9c0 con 0x7fc3140ae8f0 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc31ba09700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc31ba09700 4 set_mon_vals no callback set 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 1 -- 172.21.15.115:0/3208236068 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31000bd60 con 0x7fc3140ae8f0 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.780+0000 7fc30bfff700 10 monclient: dump: 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 10 monclient: shutdown 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 -- 172.21.15.115:0/3208236068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140ae8f0 msgr2=0x7fc3140a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 --2- 172.21.15.115:0/3208236068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140ae8f0 0x7fc3140a5d10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc314064080 tx=0x7fc310009250 comp rx=0 tx=0).stop 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 -- 172.21.15.115:0/3208236068 shutdown_connections 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 --2- 172.21.15.115:0/3208236068 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140ae8f0 0x7fc3140a5d10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 --2- 172.21.15.115:0/3208236068 >> v2:172.21.15.157:3300/0 conn(0x7fc3140aecc0 0x7fc3140b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 -- 172.21.15.115:0/3208236068 >> 172.21.15.115:0/3208236068 conn(0x7fc31406a460 msgr2=0x7fc3140630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 -- 172.21.15.115:0/3208236068 shutdown_connections 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 1 -- 172.21.15.115:0/3208236068 wait complete. 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 10 monclient: build_initial_monmap 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.781+0000 7fc31ba09700 10 monclient: monmap: 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 librados: starting msgr at 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 librados: starting objecter 2023-09-30T01:03:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 Processor -- start 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 -- start start 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 librados: setting wanted keys 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 librados: calling monclient init 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient: init 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-c con 0x7fc3140ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:39.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-a con 0x7fc3140aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:39.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3141415c0 0x7fc31414e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): picked mon.noname-b con 0x7fc3141415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:39.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140a9bc0 con 0x7fc3140aecc0 2023-09-30T01:03:39.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140ab4c0 con 0x7fc3141415c0 2023-09-30T01:03:39.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:39.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140a92c0 con 0x7fc3140ae8f0 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.782+0000 7fc31ba09700 10 monclient(hunting): authenticate will time out at 1081.232990s 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59312/0 (socket says 172.21.15.115:59312) 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:44370/0 (socket says 172.21.15.115:44370) 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 1 -- 172.21.15.115:0/1608854757 learned_addr learned my addr 172.21.15.115:0/1608854757 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): get_auth_request con 0x7fc3140ae8f0 auth_method 0 2023-09-30T01:03:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): get_auth_request con 0x7fc3140aecc0 auth_method 0 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc319fa6700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3141415c0 0x7fc31414e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc319fa6700 10 monclient(hunting): get_auth_request con 0x7fc3141415c0 auth_method 0 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc319fa6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc319fa6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:39.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc319fa6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 -- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 msgr2=0x7fc31413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 -- 172.21.15.115:0/1608854757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3141415c0 msgr2=0x7fc31414e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3141415c0 0x7fc31414e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient: _finish_hunting 0 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 monclient: found mon.noname-a 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 -- 172.21.15.115:0/1608854757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3100094f0 con 0x7fc3140aecc0 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3197a5700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:39.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc318fa4700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc304009fd0 tx=0x7fc304008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 client.?.objecter ms_handle_connect 0x7fc3140aecc0 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304013410 con 0x7fc3140aecc0 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: dump: 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 5 monclient: authenticate success, global_id 4404 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc304013570 con 0x7fc3140aecc0 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3040138f0 con 0x7fc3140aecc0 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 10 monclient: _renew_subs 2023-09-30T01:03:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31413ead0 con 0x7fc3140aecc0 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc30b7fe700 4 set_mon_vals no callback set 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc3097fa700 10 monclient: dump: 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 10 monclient: _renew_subs 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3140a9580 con 0x7fc3140aecc0 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.783+0000 7fc31ba09700 1 librados: init done 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.784+0000 7fc31ba09700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.784+0000 7fc31ba09700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.784+0000 7fc31ba09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.784+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3140a9d70 con 0x7fc3140aecc0 2023-09-30T01:03:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc304013ca0 con 0x7fc3140aecc0 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 1 --2- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 0x7fc2f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3197a5700 1 --2- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 0x7fc2f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc304083c00 con 0x7fc3140aecc0 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 10 client.4404.objecter ms_dispatch 0x7fc314001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 3 client.4404.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 3 client.4404.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3197a5700 10 monclient: get_auth_request con 0x7fc2f8068890 auth_method 0 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3097fa700 20 client.4404.objecter dump_active .. 0 homeless 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.786+0000 7fc3097fa700 10 client.4404.objecter ms_handle_connect 0x7fc2f8068890 2023-09-30T01:03:39.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.785+0000 7fc3197a5700 1 --2- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 0x7fc2f806ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc31000c010 tx=0x7fc310017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:39.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.789+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3140a9d70 con 0x7fc3140aecc0 2023-09-30T01:03:39.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.789+0000 7fc3097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:39.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:39.789+0000 7fc3097fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:40.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc31ba09700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:40.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc31ba09700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:40.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc31ba09700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:40.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc3140a9d70 con 0x7fc3140aecc0 2023-09-30T01:03:40.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc3097fa700 1 -- 172.21.15.115:0/1608854757 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc304020100 con 0x7fc3140aecc0 2023-09-30T01:03:40.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc3097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:40.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.012+0000 7fc3097fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:40.014 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:40.014 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:40.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 10 librados: watch_flush enter 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 10 librados: watch_flush exit 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 20 client.4404.objecter shutdown clearing up homeless session... 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 10 client.4404.objecter successfully canceled tick 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 msgr2=0x7fc2f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 0x7fc2f806ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc31000c010 tx=0x7fc310017000 comp rx=0 tx=0).stop 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 10 monclient: shutdown 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 msgr2=0x7fc31414e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:40.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc304009fd0 tx=0x7fc304008040 comp rx=0 tx=0).stop 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 shutdown_connections 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> v2:172.21.15.115:6816/115635 conn(0x7fc2f8068890 0x7fc2f806ad40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3141415c0 0x7fc31414e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc3140ae8f0 0x7fc31413df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 --2- 172.21.15.115:0/1608854757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3140aecc0 0x7fc31414e2a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.014+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 >> 172.21.15.115:0/1608854757 conn(0x7fc31406a460 msgr2=0x7fc3140adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.015+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 shutdown_connections 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.015+0000 7fc31ba09700 1 -- 172.21.15.115:0/1608854757 wait complete. 2023-09-30T01:03:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:40.015+0000 7fc31ba09700 1 librados: shutdown 2023-09-30T01:03:43.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:43.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.157+0000 7f1355805700 10 monclient: get_monmap_and_config 2023-09-30T01:03:43.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.157+0000 7f1355805700 10 monclient: build_initial_monmap 2023-09-30T01:03:43.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.157+0000 7f1355805700 10 monclient: monmap: 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:43.158336+0000 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:43.158336+0000 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:43.158 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:43.159 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:43.159 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.157+0000 7f1355805700 1 Processor -- start 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 -- start start 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient: init 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): picked mon.noname-b con 0x7f135006e870 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f135013dac0 0x7f135013def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): picked mon.noname-c con 0x7f135013dac0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:43.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f135013e430 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135013add0 con 0x7f135013e430 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13500aac60 con 0x7f135013dac0 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135013aad0 con 0x7f135006e870 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f1355805700 10 monclient(hunting): authenticate will time out at 1084.608684s 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:39450/0 (socket says 172.21.15.115:39450) 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 1 -- 172.21.15.115:0/2767088129 learned_addr learned my addr 172.21.15.115:0/2767088129 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:43.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): get_auth_request con 0x7f135006e870 auth_method 0 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134e7fc700 1 --2- 172.21.15.115:0/2767088129 >> v2:172.21.15.157:3300/0 conn(0x7f135013dac0 0x7f135013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134e7fc700 10 monclient(hunting): get_auth_request con 0x7f135013dac0 auth_method 0 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.158+0000 7f134e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:43.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 -- 172.21.15.115:0/2767088129 >> v2:172.21.15.157:3300/0 conn(0x7f135013dac0 msgr2=0x7f135013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 --2- 172.21.15.115:0/2767088129 >> v2:172.21.15.157:3300/0 conn(0x7f135013dac0 0x7f135013def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 -- 172.21.15.115:0/2767088129 >> v1:172.21.15.157:6789/0 conn(0x7f135013e430 legacy=0x7f135006bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 monclient: found mon.noname-b 2023-09-30T01:03:43.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 -- 172.21.15.115:0/2767088129 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135006deb0 con 0x7f135006e870 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134effd700 1 --2- 172.21.15.115:0/2767088129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13440099b0 tx=0x7f1344009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f63fc60d50cbf721 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 1 -- 172.21.15.115:0/2767088129 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344009ea0 con 0x7f135006e870 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: dump: 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:43.172 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:43.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 1 -- 172.21.15.115:0/2767088129 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134400b860 con 0x7f135006e870 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 5 monclient: authenticate success, global_id 4307 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 1 -- 172.21.15.115:0/2767088129 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400bc20 con 0x7f135006e870 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:43.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 4 set_mon_vals no callback set 2023-09-30T01:03:43.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:43.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f134dffb700 10 monclient: dump: 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:43.176 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 10 monclient: shutdown 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 1 -- 172.21.15.115:0/2767088129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 msgr2=0x7f135013a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.159+0000 7f1355805700 1 --2- 172.21.15.115:0/2767088129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13440099b0 tx=0x7f1344009250 comp rx=0 tx=0).stop 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 -- 172.21.15.115:0/2767088129 shutdown_connections 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 --2- 172.21.15.115:0/2767088129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135006e870 0x7f135013a590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 --2- 172.21.15.115:0/2767088129 >> v2:172.21.15.157:3300/0 conn(0x7f135013dac0 0x7f135013def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 -- 172.21.15.115:0/2767088129 >> 172.21.15.115:0/2767088129 conn(0x7f13500b0540 msgr2=0x7f13500b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:43.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 -- 172.21.15.115:0/2767088129 shutdown_connections 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.160+0000 7f1355805700 1 -- 172.21.15.115:0/2767088129 wait complete. 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 10 monclient: build_initial_monmap 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 10 monclient: monmap: 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:43.178 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 librados: starting msgr at 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 librados: starting objecter 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 Processor -- start 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 -- start start 2023-09-30T01:03:43.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 librados: setting wanted keys 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 1 librados: calling monclient init 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.161+0000 7f1355805700 10 monclient: init 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): picked mon.noname-a con 0x7f135006e870 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 0x7f1350135590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): picked mon.noname-c con 0x7f135013dac0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 0x7f1350135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): picked mon.noname-b con 0x7f135013e430 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350136010 con 0x7f135006e870 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500a8930 con 0x7f135013e430 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13500a9d60 con 0x7f135013dac0 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1355805700 10 monclient(hunting): authenticate will time out at 1084.612359s 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 0x7f1350135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33914/0 (socket says 172.21.15.115:33914) 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 -- 172.21.15.115:0/2502702171 learned_addr learned my addr 172.21.15.115:0/2502702171 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): get_auth_request con 0x7f135006e870 auth_method 0 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): get_auth_request con 0x7f135013dac0 auth_method 0 2023-09-30T01:03:43.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134f7fe700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 0x7f1350135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 -- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 msgr2=0x7f1350135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 0x7f1350135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 -- 172.21.15.115:0/2502702171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 msgr2=0x7f1350135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 0x7f1350135ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient: _finish_hunting 0 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 monclient: found mon.noname-a 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 -- 172.21.15.115:0/2502702171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13440094f0 con 0x7f135006e870 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134f7fe700 10 monclient: get_auth_request con 0x7f135013e430 auth_method 0 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134f7fe700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 0x7f1350135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134e7fc700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 0x7f1350135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f134effd700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1344006fd0 tx=0x7f1344017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1336ffd700 10 client.?.objecter ms_handle_connect 0x7f135006e870 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.162+0000 7f1336ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134401f070 con 0x7f135006e870 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: dump: 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1344006b10 con 0x7f135006e870 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 5 monclient: authenticate success, global_id 4407 2023-09-30T01:03:43.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _renew_subs 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1350147020 con 0x7f135006e870 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400b9d0 con 0x7f135006e870 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1336ffd700 10 monclient: dump: 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:43.190 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _renew_subs 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f134d7fa700 4 set_mon_vals no callback set 2023-09-30T01:03:43.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13500a8640 con 0x7f135006e870 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 1 librados: init done 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.163+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13500a8c10 con 0x7f135006e870 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1344023070 con 0x7f135006e870 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 1 --2- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 0x7f133c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f13440851c0 con 0x7f135006e870 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 10 client.4407.objecter ms_dispatch 0x7f1350001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 3 client.4407.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:43.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 3 client.4407.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f134e7fc700 1 --2- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 0x7f133c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f134e7fc700 10 monclient: get_auth_request con 0x7f133c068890 auth_method 0 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.164+0000 7f1336ffd700 20 client.4407.objecter dump_active .. 0 homeless 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.165+0000 7f1336ffd700 10 client.4407.objecter ms_handle_connect 0x7f133c068890 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.165+0000 7f134e7fc700 1 --2- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 0x7f133c06ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13500add90 tx=0x7f13400091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.169+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1344052ac0 con 0x7f135006e870 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.169+0000 7f1336ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:43.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.169+0000 7f1336ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1355805700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1355805700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1355805700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1350072960 con 0x7f135006e870 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1336ffd700 1 -- 172.21.15.115:0/2502702171 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1344015100 con 0x7f135006e870 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1336ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.394+0000 7f1336ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:43.395 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 10 librados: watch_flush enter 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 10 librados: watch_flush exit 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 20 client.4407.objecter shutdown clearing up homeless session... 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 10 client.4407.objecter successfully canceled tick 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 msgr2=0x7f133c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 0x7f133c06ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f13500add90 tx=0x7f13400091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 10 monclient: shutdown 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 msgr2=0x7f13500a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1344006fd0 tx=0x7f1344017000 comp rx=0 tx=0).stop 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 shutdown_connections 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> v2:172.21.15.115:6816/115635 conn(0x7f133c068890 0x7f133c06ad40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135013e430 0x7f1350135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f135006e870 0x7f13500a7950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 --2- 172.21.15.115:0/2502702171 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f135013dac0 0x7f1350135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 >> 172.21.15.115:0/2502702171 conn(0x7f13500b0540 msgr2=0x7f135006ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 shutdown_connections 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 -- 172.21.15.115:0/2502702171 wait complete. 2023-09-30T01:03:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:43.396+0000 7f1355805700 1 librados: shutdown 2023-09-30T01:03:46.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.537+0000 7f4b0b3d4700 10 monclient: get_monmap_and_config 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.537+0000 7f4b0b3d4700 10 monclient: build_initial_monmap 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.537+0000 7f4b0b3d4700 10 monclient: monmap: 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:46.538 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:46.538696+0000 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:46.538696+0000 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.538+0000 7f4b0b3d4700 1 Processor -- start 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.538+0000 7f4b0b3d4700 1 -- start start 2023-09-30T01:03:46.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.538+0000 7f4b0b3d4700 10 monclient: init 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4b040ae8f0 0x7f4b040a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-a con 0x7f4b040ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4b040aecc0 0x7f4b040b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-c con 0x7f4b040aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-b con 0x7f4b041415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b040a7040 con 0x7f4b040ae8f0 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b040b19b0 con 0x7f4b040aecc0 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040a57f0 con 0x7f4b041415c0 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient(hunting): authenticate will time out at 1087.989218s 2023-09-30T01:03:46.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4b040ae8f0 0x7f4b040a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4b040ae8f0 0x7f4b040a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33916/0 (socket says 172.21.15.115:33916) 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:39458/0 (socket says 172.21.15.115:39458) 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 -- 172.21.15.115:0/2481017618 learned_addr learned my addr 172.21.15.115:0/2481017618 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): get_auth_request con 0x7f4b040ae8f0 auth_method 0 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): get_auth_request con 0x7f4b041415c0 auth_method 0 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:46.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0896f700 1 --2- 172.21.15.115:0/2481017618 >> v2:172.21.15.157:3300/0 conn(0x7f4b040aecc0 0x7f4b040b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0896f700 10 monclient(hunting): get_auth_request con 0x7f4b040aecc0 auth_method 0 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0896f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0896f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0896f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:46.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient(hunting): handle_auth_done global_id 4410 payload 274 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 -- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 msgr2=0x7f4b040adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 -- 172.21.15.115:0/2481017618 >> v2:172.21.15.157:3300/0 conn(0x7f4b040aecc0 msgr2=0x7f4b040b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 --2- 172.21.15.115:0/2481017618 >> v2:172.21.15.157:3300/0 conn(0x7f4b040aecc0 0x7f4b040b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient: _finish_hunting 0 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 monclient: found mon.noname-a 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 -- 172.21.15.115:0/2481017618 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040a5480 con 0x7f4b040ae8f0 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09971700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b09170700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b040ae8f0 0x7f4b040a5d10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4b04064080 tx=0x7f4b00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a5228d2a6b23d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 1 -- 172.21.15.115:0/2481017618 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00009e70 con 0x7f4b040ae8f0 2023-09-30T01:03:46.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: dump: 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:46.554 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 1 -- 172.21.15.115:0/2481017618 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0000b860 con 0x7f4b040ae8f0 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 5 monclient: authenticate success, global_id 4410 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:46.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 1 -- 172.21.15.115:0/2481017618 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0000bc00 con 0x7f4b040ae8f0 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4b0b3d4700 4 set_mon_vals no callback set 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.539+0000 7f4afbfff700 10 monclient: dump: 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:46.556 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 10 monclient: shutdown 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b040ae8f0 msgr2=0x7f4b040a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b040ae8f0 0x7f4b040a5d10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4b04064080 tx=0x7f4b00009250 comp rx=0 tx=0).stop 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2481017618 shutdown_connections 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b040ae8f0 0x7f4b040a5d10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 --2- 172.21.15.115:0/2481017618 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b041415c0 0x7f4b040adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 --2- 172.21.15.115:0/2481017618 >> v2:172.21.15.157:3300/0 conn(0x7f4b040aecc0 0x7f4b040b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2481017618 >> 172.21.15.115:0/2481017618 conn(0x7f4b0406a460 msgr2=0x7f4b040630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2481017618 shutdown_connections 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.540+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2481017618 wait complete. 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 10 monclient: build_initial_monmap 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 10 monclient: monmap: 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:46.558 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 librados: starting msgr at 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 librados: starting objecter 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 Processor -- start 2023-09-30T01:03:46.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 -- start start 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 librados: setting wanted keys 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 1 librados: calling monclient init 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.541+0000 7f4b0b3d4700 10 monclient: init 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-c con 0x7f4b040ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 0x7f4b040acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-b con 0x7f4b040aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:46.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b041415c0 0x7f4b040ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): picked mon.noname-a con 0x7f4b041415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040aafc0 con 0x7f4b041415c0 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040ab450 con 0x7f4b040aecc0 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040ac3d0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient(hunting): authenticate will time out at 1087.992244s 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:39468/0 (socket says 172.21.15.115:39468) 2023-09-30T01:03:46.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 -- 172.21.15.115:0/2762445742 learned_addr learned my addr 172.21.15.115:0/2762445742 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): get_auth_request con 0x7f4b040ae8f0 auth_method 0 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b041415c0 0x7f4b040ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): get_auth_request con 0x7f4b041415c0 auth_method 0 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09971700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0896f700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 0x7f4b040acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 -- 172.21.15.115:0/2762445742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 msgr2=0x7f4b040acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 0x7f4b040acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 -- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b041415c0 msgr2=0x7f4b040ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b041415c0 0x7f4b040ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient: _finish_hunting 0 2023-09-30T01:03:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 monclient: found mon.noname-c 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 -- 172.21.15.115:0/2762445742 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc009a20 con 0x7f4b040ae8f0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0896f700 10 monclient: get_auth_request con 0x7f4b040aecc0 auth_method 0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0896f700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 0x7f4b040acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b09170700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b040b1dd0 tx=0x7f4b00017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 client.?.objecter ms_handle_connect 0x7f4b040ae8f0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0000bbe0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: dump: 2023-09-30T01:03:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: _finish_auth 0 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00011420 con 0x7f4b040ae8f0 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 5 monclient: authenticate success, global_id 4316 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient: _renew_subs 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2762445742 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b000094f0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b000117c0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4af97fa700 10 monclient: dump: 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.542+0000 7f4afb7fe700 4 set_mon_vals no callback set 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.543+0000 7f4b0b3d4700 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.543+0000 7f4b0b3d4700 10 monclient: _renew_subs 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.543+0000 7f4b0b3d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.543+0000 7f4b0b3d4700 1 -- 172.21.15.115:0/2762445742 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b040ad550 con 0x7f4b040ae8f0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.543+0000 7f4b0b3d4700 1 librados: init done 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4aeeffd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4aeeffd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4aeeffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc0050b0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4b00011bf0 con 0x7f4b040ae8f0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 1 --2- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 0x7f4ae806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4b00085790 con 0x7f4b040ae8f0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 10 client.4316.objecter ms_dispatch 0x7f4b04001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 3 client.4316.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 3 client.4316.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4b0896f700 1 --2- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 0x7f4ae806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4b0896f700 10 monclient: get_auth_request con 0x7f4ae8068890 auth_method 0 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.544+0000 7f4af97fa700 20 client.4316.objecter dump_active .. 0 homeless 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.545+0000 7f4af97fa700 10 client.4316.objecter ms_handle_connect 0x7f4ae8068890 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.545+0000 7f4b0896f700 1 --2- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 0x7f4ae806ad40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b0413bf00 tx=0x7f4af4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.549+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b00053110 con 0x7f4b040ae8f0 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.549+0000 7f4af97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:46.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.549+0000 7f4af97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.771+0000 7f4aeeffd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.771+0000 7f4aeeffd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.771+0000 7f4aeeffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.771+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4adc005360 con 0x7f4b040ae8f0 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.772+0000 7f4af97fa700 1 -- 172.21.15.115:0/2762445742 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4b00056730 con 0x7f4b040ae8f0 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.772+0000 7f4af97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.772+0000 7f4af97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:46.772 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:46.773 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 10 librados: watch_flush enter 2023-09-30T01:03:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 10 librados: watch_flush exit 2023-09-30T01:03:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 20 client.4316.objecter shutdown clearing up homeless session... 2023-09-30T01:03:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 10 client.4316.objecter successfully canceled tick 2023-09-30T01:03:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 msgr2=0x7f4ae806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 0x7f4ae806ad40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b0413bf00 tx=0x7f4af4008040 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 10 monclient: shutdown 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 msgr2=0x7f4b040aa2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.773+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b040b1dd0 tx=0x7f4b00017000 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 shutdown_connections 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> v2:172.21.15.115:6816/115635 conn(0x7f4ae8068890 0x7f4ae806ad40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4b041415c0 0x7f4b040ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4b040ae8f0 0x7f4b040aa2a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 --2- 172.21.15.115:0/2762445742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b040aecc0 0x7f4b040acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 >> 172.21.15.115:0/2762445742 conn(0x7f4b0406a460 msgr2=0x7f4b04142ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 shutdown_connections 2023-09-30T01:03:46.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 -- 172.21.15.115:0/2762445742 wait complete. 2023-09-30T01:03:46.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:46.774+0000 7f4aeeffd700 1 librados: shutdown 2023-09-30T01:03:49.786 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.914+0000 7fe0cbc4d700 10 monclient: get_monmap_and_config 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.914+0000 7fe0cbc4d700 10 monclient: build_initial_monmap 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.914+0000 7fe0cbc4d700 10 monclient: monmap: 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:49.915133+0000 2023-09-30T01:03:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:49.915133+0000 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:49.915 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.914+0000 7fe0cbc4d700 1 Processor -- start 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 1 -- start start 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient: init 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0c40ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0c40aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe0c40a5d10 0x7fe0c40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-c con 0x7fe0c40a5d10 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40a6190 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40a7040 con 0x7fe0c40aecc0 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c4141cd0 con 0x7fe0c40a5d10 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0cbc4d700 10 monclient(hunting): authenticate will time out at 1091.365723s 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0c99e9700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe0c40ae8f0 0x7fe0c40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:48758/0 (socket says 172.21.15.115:48758) 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0c99e9700 1 -- 172.21.15.115:0/2167383880 learned_addr learned my addr 172.21.15.115:0/2167383880 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.915+0000 7fe0ca1ea700 1 --2- 172.21.15.115:0/2167383880 >> v2:172.21.15.157:3300/0 conn(0x7fe0c40a5d10 0x7fe0c40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request con 0x7fe0c40a5d10 auth_method 0 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202271310 0 0) 0x7fe0b4001110 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): my global_id is 4416 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b0003570 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107626505 0 0) 0x7fe0c40a7040 con 0x7fe0c40aecc0 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): my global_id is 4423 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b4001110 con 0x7fe0c40aecc0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024641400 0 0) 0x7fe0b0003570 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 >> v2:172.21.15.157:3300/0 conn(0x7fe0c40a5d10 msgr2=0x7fe0c40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 --2- 172.21.15.115:0/2167383880 >> v2:172.21.15.157:3300/0 conn(0x7fe0c40a5d10 0x7fe0c40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 >> v1:172.21.15.157:6789/0 conn(0x7fe0c40aecc0 legacy=0x7fe0c41415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: _finish_hunting 0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40a57f0 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: _finish_auth 0 2023-09-30T01:03:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7fe0b4001600 con 0x7fe0c40ae8f0 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0cbc4d700 5 monclient: authenticate success, global_id 4416 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0cbc4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: dump: 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 >> v1:172.21.15.115:6789/0 conn(0x7fe0c40ae8f0 legacy=0x7fe0c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 0x7fe0b0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): picked mon.a con 0x7fe0b0003bf0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0004810 0x7fe0b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): picked mon.b con 0x7fe0b0004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0b00051c0 0x7fe0b000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): picked mon.c con 0x7fe0b00051c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4141cd0 con 0x7fe0b0003bf0 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b000c0b0 con 0x7fe0b0004810 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b000c270 con 0x7fe0b00051c0 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c89e7700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0cbc4d700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 0x7fe0b0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): get_auth_request con 0x7fe0b0003bf0 auth_method 0 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0b00051c0 0x7fe0b000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request con 0x7fe0b00051c0 auth_method 0 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0004810 0x7fe0b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): get_auth_request con 0x7fe0b0004810 auth_method 0 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:03:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient(hunting): handle_auth_done global_id 4416 payload 293 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 -- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0b00051c0 msgr2=0x7fe0b000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0b00051c0 0x7fe0b000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 -- 172.21.15.115:0/2167383880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0004810 msgr2=0x7fe0b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0004810 0x7fe0b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient: _finish_hunting 0 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 monclient: found mon.a 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 -- 172.21.15.115:0/2167383880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b0003570 con 0x7fe0b0003bf0 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0ca1ea700 1 -- 172.21.15.115:0/2167383880 reap_dead start 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 10 monclient: _finish_auth 0 2023-09-30T01:03:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.916+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 0x7fe0b0003fe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c440 tx=0x7fe0b4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acbc09302724739 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b400cd70 con 0x7fe0b0003bf0 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: dump: 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b4005920 con 0x7fe0b0003bf0 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 4 set_mon_vals no callback set 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 1 -- 172.21.15.115:0/2167383880 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4005ce0 con 0x7fe0b0003bf0 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0c89e7700 10 monclient: dump: 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 10 monclient: shutdown 2023-09-30T01:03:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 msgr2=0x7fe0b0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 0x7fe0b0003fe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c440 tx=0x7fe0b4007a00 comp rx=0 tx=0).stop 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/2167383880 shutdown_connections 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/2167383880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0b0003bf0 0x7fe0b0003fe0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/2167383880 >> 172.21.15.115:0/2167383880 conn(0x7fe0c406a460 msgr2=0x7fe0c40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/2167383880 shutdown_connections 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.917+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/2167383880 wait complete. 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 10 monclient: build_initial_monmap 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 10 monclient: monmap: 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:49.940 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 librados: starting msgr at 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 librados: starting objecter 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 Processor -- start 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 -- start start 2023-09-30T01:03:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 librados: setting wanted keys 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 1 librados: calling monclient init 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.918+0000 7fe0cbc4d700 10 monclient: init 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0003bf0 0x7fe0c40aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-b con 0x7fe0b0003bf0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-a con 0x7fe0c40a9520 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 0x7fe0c4150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): picked mon.noname-c con 0x7fe0c40a77d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40ab7a0 con 0x7fe0c40a9520 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40a9d30 con 0x7fe0b0003bf0 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c40a88c0 con 0x7fe0c40a77d0 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0cbc4d700 10 monclient(hunting): authenticate will time out at 1091.369337s 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 0x7fe0c4150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33952/0 (socket says 172.21.15.115:33952) 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 1 -- 172.21.15.115:0/1360734549 learned_addr learned my addr 172.21.15.115:0/1360734549 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): get_auth_request con 0x7fe0c40a9520 auth_method 0 2023-09-30T01:03:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request con 0x7fe0c40a77d0 auth_method 0 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0003bf0 0x7fe0c40aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c99e9700 10 monclient(hunting): get_auth_request con 0x7fe0b0003bf0 auth_method 0 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c99e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c99e9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c99e9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 -- 172.21.15.115:0/1360734549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0003bf0 msgr2=0x7fe0c40aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0003bf0 0x7fe0c40aa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 -- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 msgr2=0x7fe0c413e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient: _finish_hunting 0 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 monclient: found mon.noname-c 2023-09-30T01:03:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 -- 172.21.15.115:0/1360734549 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c413eda0 con 0x7fe0c40a77d0 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0c91e8700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0ca1ea700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 0x7fe0c4150f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4142650 tx=0x7fe0bc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0b9ffb700 10 client.?.objecter ms_handle_connect 0x7fe0c40a77d0 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.919+0000 7fe0b9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00e040 con 0x7fe0c40a77d0 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: dump: 2023-09-30T01:03:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc00f040 con 0x7fe0c40a77d0 2023-09-30T01:03:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 5 monclient: authenticate success, global_id 4319 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 10 monclient: _renew_subs 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c40a7c70 con 0x7fe0c40a77d0 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00fe20 con 0x7fe0c40a77d0 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0b9ffb700 10 monclient: dump: 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0bbfff700 4 set_mon_vals no callback set 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 10 monclient: _renew_subs 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c41514b0 con 0x7fe0c40a77d0 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.920+0000 7fe0cbc4d700 1 librados: init done 2023-09-30T01:03:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.921+0000 7fe0cbc4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.921+0000 7fe0cbc4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.921+0000 7fe0cbc4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.921+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c40a9f70 con 0x7fe0c40a77d0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.921+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe0bc015640 con 0x7fe0c40a77d0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 1 --2- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 0x7fe0a806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe0bc084760 con 0x7fe0c40a77d0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 10 client.4319.objecter ms_dispatch 0x7fe0c4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 3 client.4319.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 3 client.4319.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 0x7fe0a806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0c99e9700 10 monclient: get_auth_request con 0x7fe0a8068910 auth_method 0 2023-09-30T01:03:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 20 client.4319.objecter dump_active .. 0 homeless 2023-09-30T01:03:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0c99e9700 1 --2- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 0x7fe0a806adc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c440 tx=0x7fe0b4015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.922+0000 7fe0b9ffb700 10 client.4319.objecter ms_handle_connect 0x7fe0a8068910 2023-09-30T01:03:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.926+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe0c40a9f70 con 0x7fe0c40a77d0 2023-09-30T01:03:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.926+0000 7fe0b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:49.926+0000 7fe0b9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.148+0000 7fe0cbc4d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.148+0000 7fe0cbc4d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.148+0000 7fe0cbc4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:03:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.148+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe0c40a9f70 con 0x7fe0c40a77d0 2023-09-30T01:03:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.149+0000 7fe0b9ffb700 1 -- 172.21.15.115:0/1360734549 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe0c40a9f70 con 0x7fe0c40a77d0 2023-09-30T01:03:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.149+0000 7fe0b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:50.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.149+0000 7fe0b9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:50.151 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:50.151 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 10 librados: watch_flush enter 2023-09-30T01:03:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 10 librados: watch_flush exit 2023-09-30T01:03:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 20 client.4319.objecter shutdown clearing up homeless session... 2023-09-30T01:03:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 10 client.4319.objecter successfully canceled tick 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 msgr2=0x7fe0a806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 0x7fe0a806adc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400c440 tx=0x7fe0b4015000 comp rx=0 tx=0).stop 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 10 monclient: shutdown 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 msgr2=0x7fe0c4150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 0x7fe0c4150f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4142650 tx=0x7fe0bc00b040 comp rx=0 tx=0).stop 2023-09-30T01:03:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 shutdown_connections 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> v2:172.21.15.115:6816/115635 conn(0x7fe0a8068910 0x7fe0a806adc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe0c40a77d0 0x7fe0c4150f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b0003bf0 0x7fe0c40aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 --2- 172.21.15.115:0/1360734549 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0c40a9520 0x7fe0c413e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 >> 172.21.15.115:0/1360734549 conn(0x7fe0c406a460 msgr2=0x7fe0c40adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 shutdown_connections 2023-09-30T01:03:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 -- 172.21.15.115:0/1360734549 wait complete. 2023-09-30T01:03:50.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:50.151+0000 7fe0cbc4d700 1 librados: shutdown 2023-09-30T01:03:53.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.292+0000 7f8e818dc700 10 monclient: get_monmap_and_config 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.292+0000 7f8e818dc700 10 monclient: build_initial_monmap 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.292+0000 7f8e818dc700 10 monclient: monmap: 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:53.293 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:53.293820+0000 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:53.293820+0000 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:53.294 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.293+0000 7f8e818dc700 1 Processor -- start 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.293+0000 7f8e818dc700 1 -- start start 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.293+0000 7f8e818dc700 10 monclient: init 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e7c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8e7c0aecc0 0x7f8e7c0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c0aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8e7c0a5c80 0x7f8e7c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c0a5c80 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0a7040 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a5480 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a5760 con 0x7f8e7c0aecc0 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient(hunting): authenticate will time out at 1094.744119s 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8e7c0a5c80 0x7f8e7c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:53.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7affd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f8e7c0ae8f0 0x7f8e7c1415c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57372/0 (socket says 172.21.15.115:57372) 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7affd700 1 -- 172.21.15.115:0/1917973570 learned_addr learned my addr 172.21.15.115:0/1917973570 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): get_auth_request con 0x7f8e7c0a5c80 auth_method 0 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:53.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7a7fc700 1 --2- 172.21.15.115:0/1917973570 >> v2:172.21.15.157:3300/0 conn(0x7f8e7c0aecc0 0x7f8e7c0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7a7fc700 10 monclient(hunting): get_auth_request con 0x7f8e7c0aecc0 auth_method 0 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 -- 172.21.15.115:0/1917973570 >> v2:172.21.15.157:3300/0 conn(0x7f8e7c0aecc0 msgr2=0x7f8e7c0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 --2- 172.21.15.115:0/1917973570 >> v2:172.21.15.157:3300/0 conn(0x7f8e7c0aecc0 0x7f8e7c0b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 -- 172.21.15.115:0/1917973570 >> v1:172.21.15.115:6789/0 conn(0x7f8e7c0ae8f0 legacy=0x7f8e7c1415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:53.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 -- 172.21.15.115:0/1917973570 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c0b1970 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 1 -- 172.21.15.115:0/1917973570 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053033680 0 0) 0x7f8e7c0a7040 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e7b7fe700 1 --2- 172.21.15.115:0/1917973570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0a5c80 0x7f8e7c0adb60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c069b30 tx=0x7f8e74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b0549f9e481a4b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 1 -- 172.21.15.115:0/1917973570 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400e070 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: dump: 2023-09-30T01:03:53.308 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:53.309 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: _finish_auth 0 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 1 -- 172.21.15.115:0/1917973570 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7400bc30 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 5 monclient: authenticate success, global_id 4422 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 1 -- 172.21.15.115:0/1917973570 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74012430 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:53.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e818dc700 4 set_mon_vals no callback set 2023-09-30T01:03:53.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:53.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.294+0000 7f8e79ffb700 10 monclient: dump: 2023-09-30T01:03:53.311 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:53.311 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:53.311 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 10 monclient: shutdown 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:53.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 -- 172.21.15.115:0/1917973570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0a5c80 msgr2=0x7f8e7c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 --2- 172.21.15.115:0/1917973570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0a5c80 0x7f8e7c0adb60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c069b30 tx=0x7f8e74009780 comp rx=0 tx=0).stop 2023-09-30T01:03:53.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 -- 172.21.15.115:0/1917973570 shutdown_connections 2023-09-30T01:03:53.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 --2- 172.21.15.115:0/1917973570 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0a5c80 0x7f8e7c0adb60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 --2- 172.21.15.115:0/1917973570 >> v2:172.21.15.157:3300/0 conn(0x7f8e7c0aecc0 0x7f8e7c0b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 -- 172.21.15.115:0/1917973570 >> 172.21.15.115:0/1917973570 conn(0x7f8e7c06a490 msgr2=0x7f8e7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 -- 172.21.15.115:0/1917973570 shutdown_connections 2023-09-30T01:03:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.295+0000 7f8e818dc700 1 -- 172.21.15.115:0/1917973570 wait complete. 2023-09-30T01:03:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 10 monclient: build_initial_monmap 2023-09-30T01:03:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 10 monclient: monmap: 2023-09-30T01:03:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:53.315 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 librados: starting msgr at 2023-09-30T01:03:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 librados: starting objecter 2023-09-30T01:03:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 Processor -- start 2023-09-30T01:03:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 -- start start 2023-09-30T01:03:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 librados: setting wanted keys 2023-09-30T01:03:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 1 librados: calling monclient init 2023-09-30T01:03:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.296+0000 7f8e818dc700 10 monclient: init 2023-09-30T01:03:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:03:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 0x7f8e7c13e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c0a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e7c0aecc0 0x7f8e7c0ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0ab990 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a81a0 con 0x7f8e7c0aecc0 2023-09-30T01:03:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0a8360 con 0x7f8e7c0a5c80 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient(hunting): authenticate will time out at 1094.747171s 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 0x7f8e7c13e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47590/0 (socket says 172.21.15.115:47590) 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 0x7f8e7c13e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59772/0 (socket says 172.21.15.115:59772) 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 -- 172.21.15.115:0/117235756 learned_addr learned my addr 172.21.15.115:0/117235756 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): get_auth_request con 0x7f8e7c0ae8f0 auth_method 0 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:53.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): get_auth_request con 0x7f8e7c0a5c80 auth_method 0 2023-09-30T01:03:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2023-09-30T01:03:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 -- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 msgr2=0x7f8e7c13e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 0x7f8e7c13e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 -- 172.21.15.115:0/117235756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e7c0aecc0 msgr2=0x7f8e7c0ac060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e7c0aecc0 0x7f8e7c0ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 monclient: found mon.noname-a 2023-09-30T01:03:53.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 -- 172.21.15.115:0/117235756 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74009a20 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e7a7fc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00d860 tx=0x7f8e6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 client.?.objecter ms_handle_connect 0x7f8e7c0ae8f0 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:53.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0125f0 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: dump: 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:53.325 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 5 monclient: authenticate success, global_id 4428 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c012750 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:53.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c012b10 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e62ffd700 10 monclient: dump: 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.327 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:53.328 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:53.328 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:53.328 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:53.328 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:53.328 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _renew_subs 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e797fa700 4 set_mon_vals no callback set 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c0ac5a0 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _renew_subs 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c0aca10 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.297+0000 7f8e818dc700 1 librados: init done 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.298+0000 7f8e818dc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.298+0000 7f8e818dc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.298+0000 7f8e818dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.298+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c0a8720 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8e6c01e460 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 1 --2- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 0x7f8e6806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8e6c084490 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 10 client.4428.objecter ms_dispatch 0x7f8e7c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 3 client.4428.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 3 client.4428.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e7affd700 1 --2- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 0x7f8e6806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e7affd700 10 monclient: get_auth_request con 0x7f8e68068890 auth_method 0 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.299+0000 7f8e62ffd700 20 client.4428.objecter dump_active .. 0 homeless 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.300+0000 7f8e62ffd700 10 client.4428.objecter ms_handle_connect 0x7f8e68068890 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.300+0000 7f8e7affd700 1 --2- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 0x7f8e6806ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8e70007fd0 tx=0x7f8e70007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.303+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e6c051cb0 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.304+0000 7f8e62ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.304+0000 7f8e62ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:53.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e818dc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:53.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e818dc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:53.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e818dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8e7c0abad0 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e62ffd700 1 -- 172.21.15.115:0/117235756 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8e7c0abad0 con 0x7f8e7c0ae8f0 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e62ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.530+0000 7f8e62ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:53.531 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 10 librados: watch_flush enter 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 10 librados: watch_flush exit 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 20 client.4428.objecter shutdown clearing up homeless session... 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 10 client.4428.objecter successfully canceled tick 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 msgr2=0x7f8e6806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 0x7f8e6806ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8e70007fd0 tx=0x7f8e70007d30 comp rx=0 tx=0).stop 2023-09-30T01:03:53.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 10 monclient: shutdown 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 msgr2=0x7f8e7c0a7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00d860 tx=0x7f8e6c008040 comp rx=0 tx=0).stop 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 shutdown_connections 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> v2:172.21.15.115:6816/115635 conn(0x7f8e68068890 0x7f8e6806ad40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e7c0aecc0 0x7f8e7c0ac060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e7c0a5c80 0x7f8e7c13e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 --2- 172.21.15.115:0/117235756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e7c0ae8f0 0x7f8e7c0a7c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:53.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 >> 172.21.15.115:0/117235756 conn(0x7f8e7c06a490 msgr2=0x7f8e7c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:53.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 shutdown_connections 2023-09-30T01:03:53.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 -- 172.21.15.115:0/117235756 wait complete. 2023-09-30T01:03:53.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:53.532+0000 7f8e818dc700 1 librados: shutdown 2023-09-30T01:03:56.545 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.672+0000 7ff83a3ad700 10 monclient: get_monmap_and_config 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.672+0000 7ff83a3ad700 10 monclient: build_initial_monmap 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.672+0000 7ff83a3ad700 10 monclient: monmap: 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:03:56.673900+0000 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:03:56.673900+0000 2023-09-30T01:03:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:03:56.674 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.673+0000 7ff83a3ad700 1 Processor -- start 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.673+0000 7ff83a3ad700 1 -- start start 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.673+0000 7ff83a3ad700 10 monclient: init 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8340ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8340aecc0 0x7ff8340a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-a con 0x7ff8340aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-b con 0x7ff8341415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8340a7040 con 0x7ff8340ae8f0 2023-09-30T01:03:56.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8340a57f0 con 0x7ff8340aecc0 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340a5480 con 0x7ff8341415c0 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83a3ad700 10 monclient(hunting): authenticate will time out at 1098.124357s 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8340aecc0 0x7ff8340a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59778/0 (socket says 172.21.15.115:59778) 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8340aecc0 0x7ff8340a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47596/0 (socket says 172.21.15.115:47596) 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff833fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff8340ae8f0 0x7ff8340b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57378/0 (socket says 172.21.15.115:57378) 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 1 -- 172.21.15.115:0/3458488298 learned_addr learned my addr 172.21.15.115:0/3458488298 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:56.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): get_auth_request con 0x7ff8341415c0 auth_method 0 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): get_auth_request con 0x7ff8340aecc0 auth_method 0 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:56.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient(hunting): handle_auth_done global_id 4431 payload 274 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 -- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 msgr2=0x7ff8340adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 -- 172.21.15.115:0/3458488298 >> v1:172.21.15.115:6789/0 conn(0x7ff8340ae8f0 legacy=0x7ff8340b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient: _finish_hunting 0 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 monclient: found mon.noname-a 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff8337fe700 1 -- 172.21.15.115:0/3458488298 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834141a40 con 0x7ff8340aecc0 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff832ffd700 1 -- 172.21.15.115:0/3458488298 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211216703 0 0) 0x7ff8340a7040 con 0x7ff8340ae8f0 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff83894a700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.674+0000 7ff832ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff8337fe700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340aecc0 0x7ff8340a5d10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff8240098e0 tx=0x7ff8240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454af741e4334b1b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 1 -- 172.21.15.115:0/3458488298 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82400b640 con 0x7ff8340aecc0 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: dump: 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.687 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: _finish_auth 0 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:56.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 5 monclient: authenticate success, global_id 4431 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 1 -- 172.21.15.115:0/3458488298 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82400b7a0 con 0x7ff8340aecc0 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 10 monclient: get_monmap_and_config success 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 4 set_mon_vals no callback set 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 1 -- 172.21.15.115:0/3458488298 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82400bb40 con 0x7ff8340aecc0 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff832ffd700 10 monclient: dump: 2023-09-30T01:03:56.689 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 10 monclient: shutdown 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:56.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340aecc0 msgr2=0x7ff8340a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340aecc0 0x7ff8340a5d10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff8240098e0 tx=0x7ff8240091a0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3458488298 shutdown_connections 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8341415c0 0x7ff8340adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3458488298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8340aecc0 0x7ff8340a5d10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3458488298 >> 172.21.15.115:0/3458488298 conn(0x7ff83406a460 msgr2=0x7ff8340630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3458488298 shutdown_connections 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.675+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3458488298 wait complete. 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.676+0000 7ff83a3ad700 10 monclient: build_initial_monmap 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.676+0000 7ff83a3ad700 10 monclient: monmap: 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:03:56.691 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.676+0000 7ff83a3ad700 1 librados: starting msgr at 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.676+0000 7ff83a3ad700 1 librados: starting objecter 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.676+0000 7ff83a3ad700 1 Processor -- start 2023-09-30T01:03:56.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 -- start start 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 librados: setting wanted keys 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 librados: calling monclient init 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient: init 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient: _reopen_session rank -1 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 0x7ff83413e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-b con 0x7ff8340ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-c con 0x7ff8340aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 0x7ff8340aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): picked mon.noname-a con 0x7ff8341415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:56.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340a9d70 con 0x7ff8341415c0 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340ac2d0 con 0x7ff8340ae8f0 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340ab970 con 0x7ff8340aecc0 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): _renew_subs 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83a3ad700 10 monclient(hunting): authenticate will time out at 1098.127580s 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 0x7ff8340aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59780/0 (socket says 172.21.15.115:59780) 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 1 -- 172.21.15.115:0/3545019256 learned_addr learned my addr 172.21.15.115:0/3545019256 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:03:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): get_auth_request con 0x7ff8340aecc0 auth_method 0 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): get_auth_request con 0x7ff8341415c0 auth_method 0 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:03:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff833fff700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 0x7ff83413e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 -- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 msgr2=0x7ff8340abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 -- 172.21.15.115:0/3545019256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 msgr2=0x7ff83413e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 0x7ff83413e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient: _finish_hunting 0 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 monclient: found mon.noname-a 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff83894a700 1 -- 172.21.15.115:0/3545019256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009ae0 con 0x7ff8341415c0 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff833fff700 10 monclient: get_auth_request con 0x7ff8340ae8f0 auth_method 0 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff833fff700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 0x7ff83413e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.677+0000 7ff8337fe700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83894a700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 0x7ff8340aa200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009e70 tx=0x7ff82c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 client.?.objecter ms_handle_connect 0x7ff8341415c0 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c018070 con 0x7ff8341415c0 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: dump: 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: _finish_auth 0 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 5 monclient: authenticate success, global_id 4437 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00f040 con 0x7ff8341415c0 2023-09-30T01:03:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00cbb0 con 0x7ff8341415c0 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 10 monclient: _renew_subs 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff824009420 con 0x7ff8341415c0 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff81bfff700 10 monclient: dump: 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff8327fc700 4 set_mon_vals no callback set 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 10 monclient: _renew_subs 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8340a9320 con 0x7ff8341415c0 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.678+0000 7ff83a3ad700 1 librados: init done 2023-09-30T01:03:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff83a3ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff83a3ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff83a3ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8340a9ee0 con 0x7ff8341415c0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff82c003020 con 0x7ff8341415c0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.679+0000 7ff81bfff700 1 --2- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 0x7ff82006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff82c083e00 con 0x7ff8341415c0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 10 client.4437.objecter ms_dispatch 0x7ff834001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 3 client.4437.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 3 client.4437.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff833fff700 1 --2- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 0x7ff82006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff833fff700 10 monclient: get_auth_request con 0x7ff820068890 auth_method 0 2023-09-30T01:03:56.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 20 client.4437.objecter dump_active .. 0 homeless 2023-09-30T01:03:56.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff81bfff700 10 client.4437.objecter ms_handle_connect 0x7ff820068890 2023-09-30T01:03:56.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.680+0000 7ff833fff700 1 --2- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 0x7ff82006ad40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8280018d0 tx=0x7ff828009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:03:56.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.684+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff82c04d5a0 con 0x7ff8341415c0 2023-09-30T01:03:56.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.684+0000 7ff81bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:03:56.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.684+0000 7ff81bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.904+0000 7ff83a3ad700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.904+0000 7ff83a3ad700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.904+0000 7ff83a3ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.904+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff834072960 con 0x7ff8341415c0 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.905+0000 7ff81bfff700 1 -- 172.21.15.115:0/3545019256 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff82c051620 con 0x7ff8341415c0 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.905+0000 7ff81bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:03:56.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.905+0000 7ff81bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:03:56.906 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:03:56.906 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:03:56.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 10 librados: watch_flush enter 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 10 librados: watch_flush exit 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 20 client.4437.objecter shutdown clearing up homeless session... 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 10 client.4437.objecter successfully canceled tick 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 msgr2=0x7ff82006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 0x7ff82006ad40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8280018d0 tx=0x7ff828009040 comp rx=0 tx=0).stop 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 10 monclient: shutdown 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 msgr2=0x7ff8340aa200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 0x7ff8340aa200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009e70 tx=0x7ff82c00b040 comp rx=0 tx=0).stop 2023-09-30T01:03:56.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 shutdown_connections 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> v2:172.21.15.115:6816/115635 conn(0x7ff820068890 0x7ff82006ad40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8341415c0 0x7ff8340aa200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8340ae8f0 0x7ff83413e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 --2- 172.21.15.115:0/3545019256 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8340aecc0 0x7ff8340abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 >> 172.21.15.115:0/3545019256 conn(0x7ff83406a460 msgr2=0x7ff834144920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 shutdown_connections 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 -- 172.21.15.115:0/3545019256 wait complete. 2023-09-30T01:03:56.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:03:56.907+0000 7ff83a3ad700 1 librados: shutdown 2023-09-30T01:03:59.920 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.048+0000 7f58cc1fd700 10 monclient: get_monmap_and_config 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.048+0000 7f58cc1fd700 10 monclient: build_initial_monmap 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.048+0000 7f58cc1fd700 10 monclient: monmap: 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:00.049 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:00.050223+0000 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:00.050223+0000 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:00.050 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.048+0000 7f58cc1fd700 1 Processor -- start 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 -- start start 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient: init 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-b con 0x7f58c40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58c40aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f58c41415c0 0x7f58c40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-c con 0x7f58c41415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c40a7040 con 0x7f58c40aecc0 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:00.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c40a57f0 con 0x7f58c41415c0 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40a5480 con 0x7f58c40ae8f0 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58cc1fd700 10 monclient(hunting): authenticate will time out at 1101.500673s 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:59794/0 (socket says 172.21.15.115:59794) 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 1 -- 172.21.15.115:0/2587720109 learned_addr learned my addr 172.21.15.115:0/2587720109 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): get_auth_request con 0x7f58c40ae8f0 auth_method 0 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:00.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58ca79a700 1 --2- 172.21.15.115:0/2587720109 >> v2:172.21.15.157:3300/0 conn(0x7f58c41415c0 0x7f58c40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58ca79a700 10 monclient(hunting): get_auth_request con 0x7f58c41415c0 auth_method 0 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58ca79a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58ca79a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.049+0000 7f58ca79a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 -- 172.21.15.115:0/2587720109 >> v2:172.21.15.157:3300/0 conn(0x7f58c41415c0 msgr2=0x7f58c40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 --2- 172.21.15.115:0/2587720109 >> v2:172.21.15.157:3300/0 conn(0x7f58c41415c0 0x7f58c40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 -- 172.21.15.115:0/2587720109 >> v1:172.21.15.157:6789/0 conn(0x7f58c40aecc0 legacy=0x7f58c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 10 monclient: _finish_hunting 0 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 monclient: found mon.noname-b 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 -- 172.21.15.115:0/2587720109 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c40a55c0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c9f99700 1 --2- 172.21.15.115:0/2587720109 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58b40099b0 tx=0x7f58b4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=977cfeb0c07a64b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 1 -- 172.21.15.115:0/2587720109 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b400b860 con 0x7f58c40ae8f0 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: dump: 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:00.062 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: _finish_auth 0 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 5 monclient: authenticate success, global_id 4331 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:00.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 1 -- 172.21.15.115:0/2587720109 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b400b9c0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 1 -- 172.21.15.115:0/2587720109 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b400bd60 con 0x7f58c40ae8f0 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 4 set_mon_vals no callback set 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58c8f97700 10 monclient: dump: 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:00.064 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 10 monclient: shutdown 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 1 -- 172.21.15.115:0/2587720109 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 msgr2=0x7f58c40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.050+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/2587720109 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58b40099b0 tx=0x7f58b4009250 comp rx=0 tx=0).stop 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 -- 172.21.15.115:0/2587720109 shutdown_connections 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/2587720109 >> v2:172.21.15.157:3300/0 conn(0x7f58c41415c0 0x7f58c40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/2587720109 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40ae8f0 0x7f58c40a5d10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 -- 172.21.15.115:0/2587720109 >> 172.21.15.115:0/2587720109 conn(0x7f58c406a460 msgr2=0x7f58c40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 -- 172.21.15.115:0/2587720109 shutdown_connections 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 -- 172.21.15.115:0/2587720109 wait complete. 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 10 monclient: build_initial_monmap 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 10 monclient: monmap: 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.066 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 librados: starting msgr at 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.051+0000 7f58cc1fd700 1 librados: starting objecter 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 Processor -- start 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 -- start start 2023-09-30T01:04:00.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 librados: setting wanted keys 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 librados: calling monclient init 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient: init 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-a con 0x7f58c40ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:00.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40aecc0 0x7f58c4151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-c con 0x7f58c40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c41415c0 0x7f58c414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): picked mon.noname-b con 0x7f58c41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c4151450 con 0x7f58c40ae8f0 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40a7a40 con 0x7f58c41415c0 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40a97f0 con 0x7f58c40aecc0 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58cc1fd700 10 monclient(hunting): authenticate will time out at 1101.503693s 2023-09-30T01:04:00.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40aecc0 0x7f58c4151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47618/0 (socket says 172.21.15.115:47618) 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 1 -- 172.21.15.115:0/285694765 learned_addr learned my addr 172.21.15.115:0/285694765 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): get_auth_request con 0x7f58c40ae8f0 auth_method 0 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): get_auth_request con 0x7f58c40aecc0 auth_method 0 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c41415c0 0x7f58c414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): get_auth_request con 0x7f58c41415c0 auth_method 0 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9f99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58c9798700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.052+0000 7f58ca79a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 -- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40aecc0 msgr2=0x7f58c4151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40aecc0 0x7f58c4151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 -- 172.21.15.115:0/285694765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c41415c0 msgr2=0x7f58c414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c41415c0 0x7f58c414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 10 monclient: _finish_hunting 0 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 monclient: found mon.noname-a 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:00.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 -- 172.21.15.115:0/285694765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b40094f0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58c9f99700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58c40b0920 tx=0x7f58b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 client.?.objecter ms_handle_connect 0x7f58c40ae8f0 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4011e40 con 0x7f58c40ae8f0 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: dump: 2023-09-30T01:04:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b4024440 con 0x7f58c40ae8f0 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 5 monclient: authenticate success, global_id 4440 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 10 monclient: _renew_subs 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c4151eb0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4024800 con 0x7f58c40ae8f0 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58b9ffb700 10 monclient: dump: 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:00.075 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58bbfff700 4 set_mon_vals no callback set 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 10 monclient: _renew_subs 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c40a88a0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.053+0000 7f58cc1fd700 1 librados: init done 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.054+0000 7f58cc1fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.054+0000 7f58cc1fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.054+0000 7f58cc1fd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.054+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c4072960 con 0x7f58c40ae8f0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.054+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f58b401a420 con 0x7f58c40ae8f0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 1 --2- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 0x7f58a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f58b4085df0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 10 client.4440.objecter ms_dispatch 0x7f58c4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 3 client.4440.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 3 client.4440.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58c9798700 1 --2- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 0x7f58a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58c9798700 10 monclient: get_auth_request con 0x7f58a8068890 auth_method 0 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 20 client.4440.objecter dump_active .. 0 homeless 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58b9ffb700 10 client.4440.objecter ms_handle_connect 0x7f58a8068890 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.055+0000 7f58c9798700 1 --2- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 0x7f58a806ad40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f58c40a9d50 tx=0x7f58c0007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.060+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58b40536b0 con 0x7f58c40ae8f0 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.060+0000 7f58b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:00.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.060+0000 7f58b9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:00.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58cc1fd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:00.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58cc1fd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58cc1fd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58c4141a40 con 0x7f58c40ae8f0 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58b9ffb700 1 -- 172.21.15.115:0/285694765 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f58b4015100 con 0x7f58c40ae8f0 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.283+0000 7f58b9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:00.285 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 10 librados: watch_flush enter 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 10 librados: watch_flush exit 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 20 client.4440.objecter shutdown clearing up homeless session... 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 10 client.4440.objecter successfully canceled tick 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 msgr2=0x7f58a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 0x7f58a806ad40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f58c40a9d50 tx=0x7f58c0007ec0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 10 monclient: shutdown 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 msgr2=0x7f58c40a92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58c40b0920 tx=0x7f58b4017000 comp rx=0 tx=0).stop 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 shutdown_connections 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> v2:172.21.15.115:6816/115635 conn(0x7f58a8068890 0x7f58a806ad40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c41415c0 0x7f58c414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f58c40ae8f0 0x7f58c40a92b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 --2- 172.21.15.115:0/285694765 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f58c40aecc0 0x7f58c4151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 >> 172.21.15.115:0/285694765 conn(0x7f58c406a460 msgr2=0x7f58c4140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 shutdown_connections 2023-09-30T01:04:00.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 -- 172.21.15.115:0/285694765 wait complete. 2023-09-30T01:04:00.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:00.285+0000 7f58cc1fd700 1 librados: shutdown 2023-09-30T01:04:03.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.427+0000 7ff1d12da700 10 monclient: get_monmap_and_config 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.427+0000 7ff1d12da700 10 monclient: build_initial_monmap 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.427+0000 7ff1d12da700 10 monclient: monmap: 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:03.428829+0000 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:03.428829+0000 2023-09-30T01:04:03.428 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:03.429 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.428+0000 7ff1d12da700 1 Processor -- start 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.428+0000 7ff1d12da700 1 -- start start 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.428+0000 7ff1d12da700 10 monclient: init 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-b con 0x7ff1cc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:03.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff1cc0aecc0 0x7ff1cc0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-c con 0x7ff1cc0aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1cc0a5c80 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc0a7040 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0a5480 con 0x7ff1cc0aecc0 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0a5760 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient(hunting): authenticate will time out at 1104.879243s 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47244/0 (socket says 172.21.15.115:47244) 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 -- 172.21.15.115:0/1786688643 learned_addr learned my addr 172.21.15.115:0/1786688643 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): get_auth_request con 0x7ff1cc0ae8f0 auth_method 0 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 -- 172.21.15.115:0/1786688643 >> v2:172.21.15.157:3300/0 conn(0x7ff1cc0aecc0 msgr2=0x7ff1cc0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 --2- 172.21.15.115:0/1786688643 >> v2:172.21.15.157:3300/0 conn(0x7ff1cc0aecc0 0x7ff1cc0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 -- 172.21.15.115:0/1786688643 >> v1:172.21.15.157:6789/0 conn(0x7ff1cc0a5c80 legacy=0x7ff1cc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient: _finish_hunting 0 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 monclient: found mon.noname-b 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 -- 172.21.15.115:0/1786688643 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc141b00 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1caffd700 1 --2- 172.21.15.115:0/1786688643 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1c00099b0 tx=0x7ff1c0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbcd094635068fb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 1 -- 172.21.15.115:0/1786688643 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c000b820 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: dump: 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:03.441 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 5 monclient: authenticate success, global_id 4337 2023-09-30T01:04:03.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 1 -- 172.21.15.115:0/1786688643 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c000b980 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 1 -- 172.21.15.115:0/1786688643 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c000bd20 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1d12da700 4 set_mon_vals no callback set 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.429+0000 7ff1c9ffb700 10 monclient: dump: 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.443 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 10 monclient: shutdown 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 -- 172.21.15.115:0/1786688643 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 msgr2=0x7ff1cc1415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 --2- 172.21.15.115:0/1786688643 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1c00099b0 tx=0x7ff1c0009250 comp rx=0 tx=0).stop 2023-09-30T01:04:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 -- 172.21.15.115:0/1786688643 shutdown_connections 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 --2- 172.21.15.115:0/1786688643 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc1415c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 --2- 172.21.15.115:0/1786688643 >> v2:172.21.15.157:3300/0 conn(0x7ff1cc0aecc0 0x7ff1cc0b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 -- 172.21.15.115:0/1786688643 >> 172.21.15.115:0/1786688643 conn(0x7ff1cc06a490 msgr2=0x7ff1cc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 -- 172.21.15.115:0/1786688643 shutdown_connections 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.430+0000 7ff1d12da700 1 -- 172.21.15.115:0/1786688643 wait complete. 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.431+0000 7ff1d12da700 10 monclient: build_initial_monmap 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.431+0000 7ff1d12da700 10 monclient: monmap: 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:03.445 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.431+0000 7ff1d12da700 1 librados: starting msgr at 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.431+0000 7ff1d12da700 1 librados: starting objecter 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 Processor -- start 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 -- start start 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 librados: setting wanted keys 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 librados: calling monclient init 2023-09-30T01:04:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient: init 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-a con 0x7ff1cc0a5c80 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc13f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-c con 0x7ff1cc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 0x7ff1cc151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): picked mon.noname-b con 0x7ff1cc0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0a7e40 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc140340 con 0x7ff1cc0aecc0 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1404b0 con 0x7ff1cc0ae8f0 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1d12da700 10 monclient(hunting): authenticate will time out at 1104.882875s 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1ca7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc13f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56760/0 (socket says 172.21.15.115:56760) 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 1 -- 172.21.15.115:0/3102175761 learned_addr learned my addr 172.21.15.115:0/3102175761 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 10 monclient(hunting): get_auth_request con 0x7ff1cc0a5c80 auth_method 0 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1caffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.432+0000 7ff1ca7fc700 10 monclient(hunting): get_auth_request con 0x7ff1cc0ae8f0 auth_method 0 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1cb7fe700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 0x7ff1cc151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 -- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 msgr2=0x7ff1cc13f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc13f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 -- 172.21.15.115:0/3102175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 msgr2=0x7ff1cc151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 0x7ff1cc151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient: _finish_hunting 0 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 monclient: found mon.noname-a 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 -- 172.21.15.115:0/3102175761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c00094f0 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1cb7fe700 10 monclient: get_auth_request con 0x7ff1cc0aecc0 auth_method 0 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1ca7fc700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc13f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1cb7fe700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 0x7ff1cc151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1caffd700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009290 tx=0x7ff1c0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 client.?.objecter ms_handle_connect 0x7ff1cc0a5c80 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c001a460 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: dump: 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:03.451 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 5 monclient: authenticate success, global_id 4443 2023-09-30T01:04:03.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 10 monclient: _renew_subs 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c001a5c0 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc14c030 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c001a960 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1b2ffd700 10 monclient: dump: 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.453 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1c97fa700 4 set_mon_vals no callback set 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 10 monclient: _renew_subs 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc152210 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.433+0000 7ff1d12da700 1 librados: init done 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.434+0000 7ff1d12da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.434+0000 7ff1d12da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.434+0000 7ff1d12da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.434+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc072960 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff1c001ad10 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 1 --2- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 0x7ff1b806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff1c0085860 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 10 client.4443.objecter ms_dispatch 0x7ff1cc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 3 client.4443.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 3 client.4443.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1ca7fc700 1 --2- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 0x7ff1b806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:03.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1ca7fc700 10 monclient: get_auth_request con 0x7ff1b8068890 auth_method 0 2023-09-30T01:04:03.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 20 client.4443.objecter dump_active .. 0 homeless 2023-09-30T01:04:03.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1b2ffd700 10 client.4443.objecter ms_handle_connect 0x7ff1b8068890 2023-09-30T01:04:03.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.435+0000 7ff1ca7fc700 1 --2- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 0x7ff1b806ad40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0aaca0 tx=0x7ff1bc0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:03.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.439+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1c0053120 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.439+0000 7ff1b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:03.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.439+0000 7ff1b2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1d12da700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1d12da700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1d12da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1cc0b1970 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1b2ffd700 1 -- 172.21.15.115:0/3102175761 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff1c0020100 con 0x7ff1cc0a5c80 2023-09-30T01:04:03.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1b2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:03.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.662+0000 7ff1b2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:03.664 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:03.664 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:03.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 10 librados: watch_flush enter 2023-09-30T01:04:03.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 10 librados: watch_flush exit 2023-09-30T01:04:03.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 20 client.4443.objecter shutdown clearing up homeless session... 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 10 client.4443.objecter successfully canceled tick 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 msgr2=0x7ff1b806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 0x7ff1b806ad40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0aaca0 tx=0x7ff1bc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 10 monclient: shutdown 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 msgr2=0x7ff1cc0a7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009290 tx=0x7ff1c0016000 comp rx=0 tx=0).stop 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 shutdown_connections 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> v2:172.21.15.115:6816/115635 conn(0x7ff1b8068890 0x7ff1b806ad40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1cc0aecc0 0x7ff1cc151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1cc0a5c80 0x7ff1cc0a7790 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 --2- 172.21.15.115:0/3102175761 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1cc0ae8f0 0x7ff1cc13f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 >> 172.21.15.115:0/3102175761 conn(0x7ff1cc06a490 msgr2=0x7ff1cc0b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 shutdown_connections 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 -- 172.21.15.115:0/3102175761 wait complete. 2023-09-30T01:04:03.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:03.664+0000 7ff1d12da700 1 librados: shutdown 2023-09-30T01:04:06.677 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:06.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.805+0000 7faca24e2700 10 monclient: get_monmap_and_config 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.805+0000 7faca24e2700 10 monclient: build_initial_monmap 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.805+0000 7faca24e2700 10 monclient: monmap: 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:06.806638+0000 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:06.806638+0000 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.806 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:06.807 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:06.807 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:06.807 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:06.810 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:06.810 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.806+0000 7faca24e2700 1 Processor -- start 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.806+0000 7faca24e2700 1 -- start start 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.806+0000 7faca24e2700 10 monclient: init 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-a con 0x7fac9c0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac9c0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-b con 0x7fac9c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9c0a7040 con 0x7fac9c0aecc0 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c0a57f0 con 0x7fac9c0ae8f0 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0a5480 con 0x7fac9c1415c0 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient(hunting): authenticate will time out at 1108.257196s 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56768/0 (socket says 172.21.15.115:56768) 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 1 -- 172.21.15.115:0/823961614 learned_addr learned my addr 172.21.15.115:0/823961614 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:06.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): get_auth_request con 0x7fac9c0ae8f0 auth_method 0 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 --2- 172.21.15.115:0/823961614 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): get_auth_request con 0x7fac9c1415c0 auth_method 0 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:06.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 -- 172.21.15.115:0/823961614 >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 msgr2=0x7fac9c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 --2- 172.21.15.115:0/823961614 >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 -- 172.21.15.115:0/823961614 >> v1:172.21.15.157:6789/0 conn(0x7fac9c0aecc0 legacy=0x7fac9c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient: _finish_hunting 0 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 monclient: found mon.noname-b 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 -- 172.21.15.115:0/823961614 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c141a40 con 0x7fac9c1415c0 2023-09-30T01:04:06.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9bfff700 1 --2- 172.21.15.115:0/823961614 >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca0a7f700 1 --2- 172.21.15.115:0/823961614 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0adb60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fac94009e60 tx=0x7fac94009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510abcbdebb122a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 1 -- 172.21.15.115:0/823961614 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9400e070 con 0x7fac9c1415c0 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: dump: 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.823 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: _finish_auth 0 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 1 -- 172.21.15.115:0/823961614 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9400bb20 con 0x7fac9c1415c0 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 5 monclient: authenticate success, global_id 4343 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 1 -- 172.21.15.115:0/823961614 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94012430 con 0x7fac9c1415c0 2023-09-30T01:04:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7faca24e2700 4 set_mon_vals no callback set 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.807+0000 7fac9affd700 10 monclient: dump: 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 10 monclient: shutdown 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 -- 172.21.15.115:0/823961614 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 msgr2=0x7fac9c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 --2- 172.21.15.115:0/823961614 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0adb60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fac94009e60 tx=0x7fac94009780 comp rx=0 tx=0).stop 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 -- 172.21.15.115:0/823961614 shutdown_connections 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 --2- 172.21.15.115:0/823961614 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0adb60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 --2- 172.21.15.115:0/823961614 >> v2:172.21.15.115:3300/0 conn(0x7fac9c0ae8f0 0x7fac9c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 -- 172.21.15.115:0/823961614 >> 172.21.15.115:0/823961614 conn(0x7fac9c06a460 msgr2=0x7fac9c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 -- 172.21.15.115:0/823961614 shutdown_connections 2023-09-30T01:04:06.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.808+0000 7faca24e2700 1 -- 172.21.15.115:0/823961614 wait complete. 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 10 monclient: build_initial_monmap 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 10 monclient: monmap: 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:06.827 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 librados: starting msgr at 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 librados: starting objecter 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 Processor -- start 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 -- start start 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 librados: setting wanted keys 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 1 librados: calling monclient init 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.809+0000 7faca24e2700 10 monclient: init 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-a con 0x7fac9c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c0aecc0 0x7fac9c13e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-b con 0x7fac9c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0a9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): picked mon.noname-c con 0x7fac9c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0ab6b0 con 0x7fac9c0ae8f0 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0ab920 con 0x7fac9c0aecc0 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0a9790 con 0x7fac9c1415c0 2023-09-30T01:04:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:06.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 10 monclient(hunting): authenticate will time out at 1108.260297s 2023-09-30T01:04:06.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0a9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56770/0 (socket says 172.21.15.115:56770) 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 1 -- 172.21.15.115:0/669153669 learned_addr learned my addr 172.21.15.115:0/669153669 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): get_auth_request con 0x7fac9c0ae8f0 auth_method 0 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): get_auth_request con 0x7fac9c1415c0 auth_method 0 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:06.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 -- 172.21.15.115:0/669153669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c0aecc0 msgr2=0x7fac9c13e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c0aecc0 0x7fac9c13e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 -- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 msgr2=0x7fac9c0abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:06.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient: _finish_hunting 0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 monclient: found mon.noname-c 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 -- 172.21.15.115:0/669153669 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900094f0 con 0x7fac9c1415c0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9bfff700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca0a7f700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0a9250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac94015000 tx=0x7fac94018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 client.?.objecter ms_handle_connect 0x7fac9c1415c0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9401e430 con 0x7fac9c1415c0 2023-09-30T01:04:06.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: dump: 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:06.834 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:06.835 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:06.835 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: _finish_auth 0 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7faca24e2700 5 monclient: authenticate success, global_id 4346 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9401e5d0 con 0x7fac9c1415c0 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9401e990 con 0x7fac9c1415c0 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac83fff700 10 monclient: dump: 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:06.836 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 10 monclient: _renew_subs 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.810+0000 7fac9a7fc700 4 set_mon_vals no callback set 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac94009a20 con 0x7fac9c1415c0 2023-09-30T01:04:06.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 10 monclient: _renew_subs 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c0aa1f0 con 0x7fac9c1415c0 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.811+0000 7faca24e2700 1 librados: init done 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7faca24e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7faca24e2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7faca24e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c0aba60 con 0x7fac9c1415c0 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fac9401ed40 con 0x7fac9c1415c0 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 1 --2- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 0x7fac8806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:06.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac9bfff700 1 --2- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 0x7fac8806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac9bfff700 10 monclient: get_auth_request con 0x7fac88068890 auth_method 0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fac940864f0 con 0x7fac9c1415c0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 10 client.4346.objecter ms_dispatch 0x7fac9c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 3 client.4346.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.812+0000 7fac83fff700 3 client.4346.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.813+0000 7fac83fff700 20 client.4346.objecter dump_active .. 0 homeless 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.813+0000 7fac83fff700 10 client.4346.objecter ms_handle_connect 0x7fac88068890 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.813+0000 7fac9bfff700 1 --2- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 0x7fac8806ad40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac90006e80 tx=0x7fac90009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.817+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac9c0aba60 con 0x7fac9c1415c0 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.817+0000 7fac83fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:06.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:06.817+0000 7fac83fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:07.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7faca24e2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:07.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7faca24e2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:07.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7faca24e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:07.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fac9c0aba60 con 0x7fac9c1415c0 2023-09-30T01:04:07.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7fac83fff700 1 -- 172.21.15.115:0/669153669 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fac9c0aba60 con 0x7fac9c1415c0 2023-09-30T01:04:07.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7fac83fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:07.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.040+0000 7fac83fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:07.041 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:07.041 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 10 librados: watch_flush enter 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 10 librados: watch_flush exit 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 20 client.4346.objecter shutdown clearing up homeless session... 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 10 client.4346.objecter successfully canceled tick 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 msgr2=0x7fac8806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 0x7fac8806ad40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac90006e80 tx=0x7fac90009250 comp rx=0 tx=0).stop 2023-09-30T01:04:07.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 10 monclient: shutdown 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 msgr2=0x7fac9c0a9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0a9250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac94015000 tx=0x7fac94018000 comp rx=0 tx=0).stop 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 shutdown_connections 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> v2:172.21.15.115:6816/115635 conn(0x7fac88068890 0x7fac8806ad40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fac9c1415c0 0x7fac9c0a9250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fac9c0ae8f0 0x7fac9c0abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 --2- 172.21.15.115:0/669153669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c0aecc0 0x7fac9c13e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 >> 172.21.15.115:0/669153669 conn(0x7fac9c06a460 msgr2=0x7fac9c13fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 shutdown_connections 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.042+0000 7faca24e2700 1 -- 172.21.15.115:0/669153669 wait complete. 2023-09-30T01:04:07.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:07.043+0000 7faca24e2700 1 librados: shutdown 2023-09-30T01:04:10.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 10 monclient: get_monmap_and_config 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 10 monclient: build_initial_monmap 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 10 monclient: monmap: 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:10.184997+0000 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:10.184997+0000 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:10.184 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:10.185 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 1 Processor -- start 2023-09-30T01:04:10.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 1 -- start start 2023-09-30T01:04:10.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.184+0000 7f68ec99e700 10 monclient: init 2023-09-30T01:04:10.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:10.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f68e80ae8f0 0x7f68e80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-c con 0x7f68e80ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68e80aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68e81415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80a7040 con 0x7f68e80aecc0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80a57f0 con 0x7f68e81415c0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80a5480 con 0x7f68e80ae8f0 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68ec99e700 10 monclient(hunting): authenticate will time out at 1111.635538s 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e5d9b700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f68e80aecc0 0x7f68e80b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:34564/0 (socket says 172.21.15.115:34564) 2023-09-30T01:04:10.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e5d9b700 1 -- 172.21.15.115:0/3433043953 learned_addr learned my addr 172.21.15.115:0/3433043953 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 1 --2- 172.21.15.115:0/3433043953 >> v2:172.21.15.157:3300/0 conn(0x7f68e80ae8f0 0x7f68e80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): get_auth_request con 0x7f68e80ae8f0 auth_method 0 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185967007 0 0) 0x7f68e80a7040 con 0x7f68e80aecc0 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e559a700 10 monclient(hunting): my global_id is 4452 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d0003570 con 0x7f68e80aecc0 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:10.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.185+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458380040 0 0) 0x7f68e80a57f0 con 0x7f68e81415c0 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient(hunting): my global_id is 4444 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68e80a7040 con 0x7f68e81415c0 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 10 monclient(hunting): handle_auth_done global_id 4441 payload 274 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1646262712 0 0) 0x7f68d0003570 con 0x7f68e80aecc0 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 1 -- 172.21.15.115:0/3433043953 >> v1:172.21.15.157:6789/0 conn(0x7f68e81415c0 legacy=0x7f68e80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 1 -- 172.21.15.115:0/3433043953 >> v1:172.21.15.115:6789/0 conn(0x7f68e80aecc0 legacy=0x7f68e80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:10.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 1 monclient: found mon.noname-c 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 1 -- 172.21.15.115:0/3433043953 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80a55c0 con 0x7f68e80ae8f0 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e659c700 1 --2- 172.21.15.115:0/3433043953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80ae8f0 0x7f68e80a5d10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0098e0 tx=0x7f68dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512bd01ee22b5fd2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8001470 con 0x7f68e80ae8f0 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: dump: 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:10.199 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: _finish_auth 0 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc011420 con 0x7f68e80ae8f0 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68ec99e700 5 monclient: authenticate success, global_id 4441 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68ec99e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:10.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68ec99e700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:10.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 1 -- 172.21.15.115:0/3433043953 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0117a0 con 0x7f68e80ae8f0 2023-09-30T01:04:10.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:10.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68ec99e700 4 set_mon_vals no callback set 2023-09-30T01:04:10.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.186+0000 7f68e559a700 10 monclient: dump: 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 10 monclient: shutdown 2023-09-30T01:04:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 -- 172.21.15.115:0/3433043953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80ae8f0 msgr2=0x7f68e80a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 --2- 172.21.15.115:0/3433043953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80ae8f0 0x7f68e80a5d10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0098e0 tx=0x7f68dc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 -- 172.21.15.115:0/3433043953 shutdown_connections 2023-09-30T01:04:10.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 --2- 172.21.15.115:0/3433043953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80ae8f0 0x7f68e80a5d10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 -- 172.21.15.115:0/3433043953 >> 172.21.15.115:0/3433043953 conn(0x7f68e806a460 msgr2=0x7f68e80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 -- 172.21.15.115:0/3433043953 shutdown_connections 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 1 -- 172.21.15.115:0/3433043953 wait complete. 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 10 monclient: build_initial_monmap 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.187+0000 7f68ec99e700 10 monclient: monmap: 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 librados: starting msgr at 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 librados: starting objecter 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 Processor -- start 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 -- start start 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 librados: setting wanted keys 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 librados: calling monclient init 2023-09-30T01:04:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient: init 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-c con 0x7f68e80ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 0x7f68e813e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-b con 0x7f68e80aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): picked mon.noname-a con 0x7f68e81415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80ab500 con 0x7f68e81415c0 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80ab7e0 con 0x7f68e80aecc0 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80abac0 con 0x7f68e80ae8f0 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68ec99e700 10 monclient(hunting): authenticate will time out at 1111.638737s 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47274/0 (socket says 172.21.15.115:47274) 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56778/0 (socket says 172.21.15.115:56778) 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 1 -- 172.21.15.115:0/2419334369 learned_addr learned my addr 172.21.15.115:0/2419334369 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): get_auth_request con 0x7f68e80ae8f0 auth_method 0 2023-09-30T01:04:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): get_auth_request con 0x7f68e81415c0 auth_method 0 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:10.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.188+0000 7f68e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 0x7f68e813e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 10 monclient(hunting): get_auth_request con 0x7f68e80aecc0 auth_method 0 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 -- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 msgr2=0x7f68e814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 -- 172.21.15.115:0/2419334369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 msgr2=0x7f68e813e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 0x7f68e813e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:04:10.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 monclient: found mon.noname-a 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 -- 172.21.15.115:0/2419334369 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc009420 con 0x7f68e81415c0 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e5d9b700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 0x7f68e813e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e659c700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e6d9d700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68e000c8e0 tx=0x7f68e00026f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 client.?.objecter ms_handle_connect 0x7f68e81415c0 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0012630 con 0x7f68e81415c0 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:10.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: dump: 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: _finish_auth 0 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:10.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e00127b0 con 0x7f68e81415c0 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 5 monclient: authenticate success, global_id 4455 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 10 monclient: _renew_subs 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e80a9930 con 0x7f68e81415c0 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0012b70 con 0x7f68e81415c0 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68d67fc700 10 monclient: dump: 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:10.213 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68e4d99700 4 set_mon_vals no callback set 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 10 monclient: _renew_subs 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e80aa6d0 con 0x7f68e81415c0 2023-09-30T01:04:10.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.189+0000 7f68ec99e700 1 librados: init done 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.190+0000 7f68ec99e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.190+0000 7f68ec99e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.190+0000 7f68ec99e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.190+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e8072960 con 0x7f68e81415c0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f68e001b400 con 0x7f68e81415c0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 1 --2- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 0x7f68c806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f68e0084770 con 0x7f68e81415c0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68e659c700 1 --2- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 0x7f68c806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 10 client.4455.objecter ms_dispatch 0x7f68e8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 3 client.4455.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 3 client.4455.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:10.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68e659c700 10 monclient: get_auth_request con 0x7f68c8068890 auth_method 0 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.191+0000 7f68d67fc700 20 client.4455.objecter dump_active .. 0 homeless 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.192+0000 7f68d67fc700 10 client.4455.objecter ms_handle_connect 0x7f68c8068890 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.192+0000 7f68e659c700 1 --2- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 0x7f68c806ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68dc003080 tx=0x7f68dc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.195+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f68e0052010 con 0x7f68e81415c0 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.195+0000 7f68d67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:10.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.195+0000 7f68d67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:10.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68ec99e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:10.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68ec99e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68ec99e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f68e8141a40 con 0x7f68e81415c0 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68d67fc700 1 -- 172.21.15.115:0/2419334369 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f68e0055630 con 0x7f68e81415c0 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68d67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.418+0000 7f68d67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:10.419 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 10 librados: watch_flush enter 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 10 librados: watch_flush exit 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 20 client.4455.objecter shutdown clearing up homeless session... 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 10 client.4455.objecter successfully canceled tick 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 msgr2=0x7f68c806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 0x7f68c806ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68dc003080 tx=0x7f68dc016000 comp rx=0 tx=0).stop 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 10 monclient: shutdown 2023-09-30T01:04:10.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 msgr2=0x7f68e80aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68e000c8e0 tx=0x7f68e00026f0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 shutdown_connections 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> v2:172.21.15.115:6816/115635 conn(0x7f68c8068890 0x7f68c806ad40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68e81415c0 0x7f68e80aa190 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f68e80ae8f0 0x7f68e814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 --2- 172.21.15.115:0/2419334369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68e80aecc0 0x7f68e813e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 >> 172.21.15.115:0/2419334369 conn(0x7f68e806a460 msgr2=0x7f68e813abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:10.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 shutdown_connections 2023-09-30T01:04:10.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 -- 172.21.15.115:0/2419334369 wait complete. 2023-09-30T01:04:10.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:10.420+0000 7f68ec99e700 1 librados: shutdown 2023-09-30T01:04:13.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.558+0000 7fa60110a700 10 monclient: get_monmap_and_config 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.558+0000 7fa60110a700 10 monclient: build_initial_monmap 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.558+0000 7fa60110a700 10 monclient: monmap: 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:13.559374+0000 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:13.559374+0000 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:13.575 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.558+0000 7fa60110a700 1 Processor -- start 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 -- start start 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient: init 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5fc0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:13.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa5fc0aecc0 0x7fa5fc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa5fc1415c0 0x7fa5fc0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0a7040 con 0x7fa5fc0ae8f0 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc0a57f0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc0a5480 con 0x7fa5fc1415c0 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa60110a700 10 monclient(hunting): authenticate will time out at 1115.009681s 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa5fc0aecc0 0x7fa5fc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fad9d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa5fc0ae8f0 0x7fa5fc0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52576/0 (socket says 172.21.15.115:52576) 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fad9d700 1 -- 172.21.15.115:0/2549599776 learned_addr learned my addr 172.21.15.115:0/2549599776 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): get_auth_request con 0x7fa5fc0aecc0 auth_method 0 2023-09-30T01:04:13.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.559+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2023-09-30T01:04:13.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 -- 172.21.15.115:0/2549599776 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc1415c0 msgr2=0x7fa5fc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 --2- 172.21.15.115:0/2549599776 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc1415c0 0x7fa5fc0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 -- 172.21.15.115:0/2549599776 >> v1:172.21.15.115:6789/0 conn(0x7fa5fc0ae8f0 legacy=0x7fa5fc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 monclient: found mon.noname-a 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 1 -- 172.21.15.115:0/2549599776 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469633984 0 0) 0x7fa5fc0a7040 con 0x7fa5fc0ae8f0 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 -- 172.21.15.115:0/2549599776 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0a55c0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:13.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5fa59c700 1 --2- 172.21.15.115:0/2549599776 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a5d10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0a6250 tx=0x7fa5ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476d8459b065ad2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 1 -- 172.21.15.115:0/2549599776 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00b6b0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: dump: 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: _finish_auth 0 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 5 monclient: authenticate success, global_id 4458 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 1 -- 172.21.15.115:0/2549599776 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec00b810 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 4 set_mon_vals no callback set 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 1 -- 172.21.15.115:0/2549599776 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00bbb0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa5f9d9b700 10 monclient: dump: 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:13.583 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 10 monclient: shutdown 2023-09-30T01:04:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 -- 172.21.15.115:0/2549599776 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 msgr2=0x7fa5fc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 --2- 172.21.15.115:0/2549599776 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a5d10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0a6250 tx=0x7fa5ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 -- 172.21.15.115:0/2549599776 shutdown_connections 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 --2- 172.21.15.115:0/2549599776 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc1415c0 0x7fa5fc0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 --2- 172.21.15.115:0/2549599776 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a5d10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 -- 172.21.15.115:0/2549599776 >> 172.21.15.115:0/2549599776 conn(0x7fa5fc06a460 msgr2=0x7fa5fc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 -- 172.21.15.115:0/2549599776 shutdown_connections 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.560+0000 7fa60110a700 1 -- 172.21.15.115:0/2549599776 wait complete. 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.561+0000 7fa60110a700 10 monclient: build_initial_monmap 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.561+0000 7fa60110a700 10 monclient: monmap: 2023-09-30T01:04:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.561+0000 7fa60110a700 1 librados: starting msgr at 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.561+0000 7fa60110a700 1 librados: starting objecter 2023-09-30T01:04:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 Processor -- start 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 -- start start 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 librados: setting wanted keys 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 librados: calling monclient init 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient: init 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc0ae8f0 0x7fa5fc0a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:13.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa5fc1415c0 0x7fa5fc14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc14e060 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0a9c60 con 0x7fa5fc0ae8f0 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0aaa70 con 0x7fa5fc1415c0 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa60110a700 10 monclient(hunting): authenticate will time out at 1115.012751s 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37026/0 (socket says 172.21.15.115:37026) 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 1 -- 172.21.15.115:0/1983474513 learned_addr learned my addr 172.21.15.115:0/1983474513 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:13.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): get_auth_request con 0x7fa5fc0aecc0 auth_method 0 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa5fc1415c0 0x7fa5fc14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): get_auth_request con 0x7fa5fc1415c0 auth_method 0 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.562+0000 7fa5fb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fad9d700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc0ae8f0 0x7fa5fc0a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fad9d700 10 monclient(hunting): get_auth_request con 0x7fa5fc0ae8f0 auth_method 0 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:13.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 -- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa5fc1415c0 msgr2=0x7fa5fc14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa5fc1415c0 0x7fa5fc14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 -- 172.21.15.115:0/1983474513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc0ae8f0 msgr2=0x7fa5fc0a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc0ae8f0 0x7fa5fc0a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 monclient: found mon.noname-a 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 -- 172.21.15.115:0/1983474513 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009420 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5fa59c700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0b0920 tx=0x7fa5ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:13.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 client.?.objecter ms_handle_connect 0x7fa5fc0aecc0 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00bb90 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: dump: 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.592 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec011420 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 5 monclient: authenticate success, global_id 4464 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 10 monclient: _renew_subs 2023-09-30T01:04:13.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc0a8d80 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0117c0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5e2ffd700 10 monclient: dump: 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:13.594 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa5f959a700 4 set_mon_vals no callback set 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 10 monclient: _renew_subs 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc13d9f0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.563+0000 7fa60110a700 1 librados: init done 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.564+0000 7fa60110a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.564+0000 7fa60110a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.564+0000 7fa60110a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.564+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc072960 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.564+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa5ec011b30 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 1 --2- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 0x7fa5e806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa5ec0858a0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 10 client.4464.objecter ms_dispatch 0x7fa5fc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 3 client.4464.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 3 client.4464.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5fad9d700 1 --2- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 0x7fa5e806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5fad9d700 10 monclient: get_auth_request con 0x7fa5e8068890 auth_method 0 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 20 client.4464.objecter dump_active .. 0 homeless 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5e2ffd700 10 client.4464.objecter ms_handle_connect 0x7fa5e8068890 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.565+0000 7fa5fad9d700 1 --2- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 0x7fa5e806ad40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00011e0 tx=0x7fa5f0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.569+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5ec053160 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.569+0000 7fa5e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:13.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.569+0000 7fa5e2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:13.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.792+0000 7fa60110a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa60110a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa60110a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa5fc141a40 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa5e2ffd700 1 -- 172.21.15.115:0/1983474513 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa5ec006cb0 con 0x7fa5fc0aecc0 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa5e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:13.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.793+0000 7fa5e2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:13.795 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:13.795 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 10 librados: watch_flush enter 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 10 librados: watch_flush exit 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 20 client.4464.objecter shutdown clearing up homeless session... 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 10 client.4464.objecter successfully canceled tick 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 msgr2=0x7fa5e806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 0x7fa5e806ad40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00011e0 tx=0x7fa5f0009040 comp rx=0 tx=0).stop 2023-09-30T01:04:13.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 10 monclient: shutdown 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 msgr2=0x7fa5fc0a8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0b0920 tx=0x7fa5ec017000 comp rx=0 tx=0).stop 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 shutdown_connections 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> v2:172.21.15.115:6816/115635 conn(0x7fa5e8068890 0x7fa5e806ad40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa5fc1415c0 0x7fa5fc14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc0ae8f0 0x7fa5fc0a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 --2- 172.21.15.115:0/1983474513 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5fc0aecc0 0x7fa5fc0a8840 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:13.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 >> 172.21.15.115:0/1983474513 conn(0x7fa5fc06a460 msgr2=0x7fa5fc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:13.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 shutdown_connections 2023-09-30T01:04:13.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 -- 172.21.15.115:0/1983474513 wait complete. 2023-09-30T01:04:13.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:13.795+0000 7fa60110a700 1 librados: shutdown 2023-09-30T01:04:16.808 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:16.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.936+0000 7f4432c10700 10 monclient: get_monmap_and_config 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.936+0000 7f4432c10700 10 monclient: build_initial_monmap 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.936+0000 7f4432c10700 10 monclient: monmap: 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:16.937630+0000 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:16.937630+0000 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:16.937 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:16.938 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:16.938 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.937+0000 7f4432c10700 1 Processor -- start 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.937+0000 7f4432c10700 1 -- start start 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.937+0000 7f4432c10700 10 monclient: init 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.937+0000 7f4432c10700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f442c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 0x7f442c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-b con 0x7f442c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f442c1415c0 0x7f442c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-a con 0x7f442c1415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c0a7040 con 0x7f442c0ae8f0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442c0a57f0 con 0x7f442c1415c0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0a5480 con 0x7f442c0aecc0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient(hunting): authenticate will time out at 1118.388088s 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 0x7f442c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f442c1415c0 0x7f442c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44309ac700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f442c0ae8f0 0x7f442c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52590/0 (socket says 172.21.15.115:52590) 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): get_auth_request con 0x7f442c1415c0 auth_method 0 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44309ac700 1 -- 172.21.15.115:0/1629269390 learned_addr learned my addr 172.21.15.115:0/1629269390 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): get_auth_request con 0x7f442c0aecc0 auth_method 0 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:16.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient(hunting): handle_auth_done global_id 4467 payload 274 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 -- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 msgr2=0x7f442c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 0x7f442c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 -- 172.21.15.115:0/1629269390 >> v1:172.21.15.115:6789/0 conn(0x7f442c0ae8f0 legacy=0x7f442c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient: _finish_hunting 0 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 monclient: found mon.noname-a 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:16.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 -- 172.21.15.115:0/1629269390 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0a55c0 con 0x7f442c1415c0 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 1 -- 172.21.15.115:0/1629269390 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477255971 0 0) 0x7f442c0a7040 con 0x7f442c0ae8f0 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442bfff700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 0x7f442c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f44311ad700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c1415c0 0x7f442c0adb60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4424009e60 tx=0x7f4424009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5987a4324a6fb8a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 1 -- 172.21.15.115:0/1629269390 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400e070 con 0x7f442c1415c0 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: dump: 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.950 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 1 -- 172.21.15.115:0/1629269390 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442400bb60 con 0x7f442c1415c0 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 5 monclient: authenticate success, global_id 4467 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:16.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 1 -- 172.21.15.115:0/1629269390 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424012430 con 0x7f442c1415c0 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f4432c10700 4 set_mon_vals no callback set 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.938+0000 7f442b7fe700 10 monclient: dump: 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:16.952 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 10 monclient: shutdown 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 -- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c1415c0 msgr2=0x7f442c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c1415c0 0x7f442c0adb60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4424009e60 tx=0x7f4424009780 comp rx=0 tx=0).stop 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 -- 172.21.15.115:0/1629269390 shutdown_connections 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c1415c0 0x7f442c0adb60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 --2- 172.21.15.115:0/1629269390 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c0aecc0 0x7f442c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:16.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 -- 172.21.15.115:0/1629269390 >> 172.21.15.115:0/1629269390 conn(0x7f442c06a460 msgr2=0x7f442c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 -- 172.21.15.115:0/1629269390 shutdown_connections 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.939+0000 7f4432c10700 1 -- 172.21.15.115:0/1629269390 wait complete. 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.940+0000 7f4432c10700 10 monclient: build_initial_monmap 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.940+0000 7f4432c10700 10 monclient: monmap: 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:16.954 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.940+0000 7f4432c10700 1 librados: starting msgr at 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.940+0000 7f4432c10700 1 librados: starting objecter 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.940+0000 7f4432c10700 1 Processor -- start 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 -- start start 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 librados: setting wanted keys 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 librados: calling monclient init 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient: init 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-a con 0x7f442c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 0x7f442c151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-b con 0x7f442c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 0x7f442c14dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): picked mon.noname-c con 0x7f442c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0a9920 con 0x7f442c0ae8f0 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c13eec0 con 0x7f442c0aecc0 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c151ea0 con 0x7f442c1415c0 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f4432c10700 10 monclient(hunting): authenticate will time out at 1118.391720s 2023-09-30T01:04:16.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37048/0 (socket says 172.21.15.115:37048) 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 1 -- 172.21.15.115:0/3117649774 learned_addr learned my addr 172.21.15.115:0/3117649774 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): get_auth_request con 0x7f442c0ae8f0 auth_method 0 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 0x7f442c14dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): get_auth_request con 0x7f442c1415c0 auth_method 0 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:16.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44309ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.941+0000 7f44311ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f442bfff700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 0x7f442c151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 -- 172.21.15.115:0/3117649774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 msgr2=0x7f442c151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 0x7f442c151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 -- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 msgr2=0x7f442c0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 10 monclient: _finish_hunting 0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 monclient: found mon.noname-c 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 -- 172.21.15.115:0/3117649774 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009420 con 0x7f442c1415c0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f442bfff700 10 monclient: get_auth_request con 0x7f442c0aecc0 auth_method 0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44309ac700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f442bfff700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 0x7f442c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f44311ad700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 0x7f442c14dd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4424016000 tx=0x7f4424019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 client.?.objecter ms_handle_connect 0x7f442c1415c0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424027470 con 0x7f442c1415c0 2023-09-30T01:04:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: dump: 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: _finish_auth 0 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 5 monclient: authenticate success, global_id 4358 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4424027610 con 0x7f442c1415c0 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 10 monclient: _renew_subs 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4424009a20 con 0x7f442c1415c0 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44240279d0 con 0x7f442c1415c0 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4428ff9700 10 monclient: dump: 2023-09-30T01:04:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f442affd700 4 set_mon_vals no callback set 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 10 monclient: _renew_subs 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c14e310 con 0x7f442c1415c0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.942+0000 7f4432c10700 1 librados: init done 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.943+0000 7f4432c10700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.943+0000 7f4432c10700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.943+0000 7f4432c10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.943+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442c14dba0 con 0x7f442c1415c0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.943+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4424024070 con 0x7f442c1415c0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 1 --2- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 0x7f441006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f44240864f0 con 0x7f442c1415c0 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 10 client.4358.objecter ms_dispatch 0x7f442c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 3 client.4358.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f44309ac700 1 --2- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 0x7f441006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 3 client.4358.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f44309ac700 10 monclient: get_auth_request con 0x7f4410068890 auth_method 0 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 20 client.4358.objecter dump_active .. 0 homeless 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f4428ff9700 10 client.4358.objecter ms_handle_connect 0x7f4410068890 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.944+0000 7f44309ac700 1 --2- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 0x7f441006ad40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4420007fd0 tx=0x7f4420007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.948+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4424053db0 con 0x7f442c1415c0 2023-09-30T01:04:16.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.948+0000 7f4428ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:16.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:16.948+0000 7f4428ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.169+0000 7f4432c10700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.169+0000 7f4432c10700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.169+0000 7f4432c10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.169+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f442c0b18d0 con 0x7f442c1415c0 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.170+0000 7f4428ff9700 1 -- 172.21.15.115:0/3117649774 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f44240573d0 con 0x7f442c1415c0 2023-09-30T01:04:17.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.170+0000 7f4428ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:17.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.170+0000 7f4428ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:17.171 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:17.171 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:17.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.171+0000 7f4432c10700 10 librados: watch_flush enter 2023-09-30T01:04:17.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.171+0000 7f4432c10700 10 librados: watch_flush exit 2023-09-30T01:04:17.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.171+0000 7f4432c10700 20 client.4358.objecter shutdown clearing up homeless session... 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.171+0000 7f4432c10700 10 client.4358.objecter successfully canceled tick 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 msgr2=0x7f441006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 0x7f441006ad40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4420007fd0 tx=0x7f4420007df0 comp rx=0 tx=0).stop 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 10 monclient: shutdown 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 msgr2=0x7f442c14dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:17.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 0x7f442c14dd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4424016000 tx=0x7f4424019000 comp rx=0 tx=0).stop 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 shutdown_connections 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> v2:172.21.15.115:6816/115635 conn(0x7f4410068890 0x7f441006ad40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f442c1415c0 0x7f442c14dd50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f442c0ae8f0 0x7f442c0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 --2- 172.21.15.115:0/3117649774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f442c0aecc0 0x7f442c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 >> 172.21.15.115:0/3117649774 conn(0x7f442c06a460 msgr2=0x7f442c13a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 shutdown_connections 2023-09-30T01:04:17.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 -- 172.21.15.115:0/3117649774 wait complete. 2023-09-30T01:04:17.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:17.172+0000 7f4432c10700 1 librados: shutdown 2023-09-30T01:04:20.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.312+0000 7f3444a0a700 10 monclient: get_monmap_and_config 2023-09-30T01:04:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.312+0000 7f3444a0a700 10 monclient: build_initial_monmap 2023-09-30T01:04:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.312+0000 7f3444a0a700 10 monclient: monmap: 2023-09-30T01:04:20.313 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:20.313883+0000 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:20.313883+0000 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:20.314 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:20.315 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:20.315 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.313+0000 7f3444a0a700 1 Processor -- start 2023-09-30T01:04:20.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.313+0000 7f3444a0a700 1 -- start start 2023-09-30T01:04:20.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.313+0000 7f3444a0a700 10 monclient: init 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f34400a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-b con 0x7f34400ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34400aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34401415c0 0x7f34400adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-c con 0x7f34401415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34400a7040 con 0x7f34400aecc0 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400a57f0 con 0x7f34401415c0 2023-09-30T01:04:20.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34400a5480 con 0x7f34400ae8f0 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f3444a0a700 10 monclient(hunting): authenticate will time out at 1121.764364s 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f34400a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343dd9b700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f34400aecc0 0x7f34400b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52598/0 (socket says 172.21.15.115:52598) 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343dd9b700 1 -- 172.21.15.115:0/1702866704 learned_addr learned my addr 172.21.15.115:0/1702866704 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): get_auth_request con 0x7f34400ae8f0 auth_method 0 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:20.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient(hunting): handle_auth_done global_id 4361 payload 274 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343d59a700 1 -- 172.21.15.115:0/1702866704 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037173952 0 0) 0x7f34400a7040 con 0x7f34400aecc0 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 -- 172.21.15.115:0/1702866704 >> v2:172.21.15.157:3300/0 conn(0x7f34401415c0 msgr2=0x7f34400adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 --2- 172.21.15.115:0/1702866704 >> v2:172.21.15.157:3300/0 conn(0x7f34401415c0 0x7f34400adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 -- 172.21.15.115:0/1702866704 >> v1:172.21.15.115:6789/0 conn(0x7f34400aecc0 legacy=0x7f34400b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 monclient: found mon.noname-b 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:20.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 -- 172.21.15.115:0/1702866704 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400a55c0 con 0x7f34400ae8f0 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343d59a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.314+0000 7f343e59c700 1 --2- 172.21.15.115:0/1702866704 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f34400a5d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f34340098e0 tx=0x7f34340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=182b74b3b948327a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 1 -- 172.21.15.115:0/1702866704 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434009e80 con 0x7f34400ae8f0 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: dump: 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:20.326 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: _finish_auth 0 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 5 monclient: authenticate success, global_id 4361 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 1 -- 172.21.15.115:0/1702866704 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400b860 con 0x7f34400ae8f0 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:20.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 4 set_mon_vals no callback set 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 1 -- 172.21.15.115:0/1702866704 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400bbe0 con 0x7f34400ae8f0 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f343d59a700 10 monclient: dump: 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:20.328 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 10 monclient: shutdown 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 -- 172.21.15.115:0/1702866704 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 msgr2=0x7f34400a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 --2- 172.21.15.115:0/1702866704 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f34400a5d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f34340098e0 tx=0x7f34340091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 -- 172.21.15.115:0/1702866704 shutdown_connections 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 --2- 172.21.15.115:0/1702866704 >> v2:172.21.15.157:3300/0 conn(0x7f34401415c0 0x7f34400adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 --2- 172.21.15.115:0/1702866704 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f34400a5d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 -- 172.21.15.115:0/1702866704 >> 172.21.15.115:0/1702866704 conn(0x7f344006a460 msgr2=0x7f34400630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:20.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 -- 172.21.15.115:0/1702866704 shutdown_connections 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.315+0000 7f3444a0a700 1 -- 172.21.15.115:0/1702866704 wait complete. 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 10 monclient: build_initial_monmap 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 10 monclient: monmap: 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:20.330 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 librados: starting msgr at 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 librados: starting objecter 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 Processor -- start 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 -- start start 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 librados: setting wanted keys 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 1 librados: calling monclient init 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.316+0000 7f3444a0a700 10 monclient: init 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-c con 0x7f34400ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-a con 0x7f34400aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 0x7f34400abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): picked mon.noname-b con 0x7f34401415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34400ab690 con 0x7f34400aecc0 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34401519a0 con 0x7f34401415c0 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440151c50 con 0x7f34400ae8f0 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:20.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient(hunting): authenticate will time out at 1121.767213s 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:37654/0 (socket says 172.21.15.115:37654) 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37060/0 (socket says 172.21.15.115:37060) 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 1 -- 172.21.15.115:0/4289560675 learned_addr learned my addr 172.21.15.115:0/4289560675 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): get_auth_request con 0x7f34400ae8f0 auth_method 0 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): get_auth_request con 0x7f34400aecc0 auth_method 0 2023-09-30T01:04:20.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343ed9d700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 0x7f34400abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 -- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 msgr2=0x7f344013f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 -- 172.21.15.115:0/4289560675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 msgr2=0x7f34400abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 0x7f34400abcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 monclient: found mon.noname-a 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 -- 172.21.15.115:0/4289560675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009420 con 0x7f34400aecc0 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343ed9d700 10 monclient: get_auth_request con 0x7f34401415c0 auth_method 0 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343e59c700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343ed9d700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 0x7f34400abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:04:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343dd9b700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f34300011e0 tx=0x7f3430007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 client.?.objecter ms_handle_connect 0x7f34400aecc0 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430005b30 con 0x7f34400aecc0 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: dump: 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.336 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3430005c90 con 0x7f34400aecc0 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 5 monclient: authenticate success, global_id 4479 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _renew_subs 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34401400d0 con 0x7f34400aecc0 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430007320 con 0x7f34400aecc0 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f342e7fc700 10 monclient: dump: 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f343cd99700 4 set_mon_vals no callback set 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _renew_subs 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34400ac560 con 0x7f34400aecc0 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.317+0000 7f3444a0a700 1 librados: init done 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.318+0000 7f3444a0a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.318+0000 7f3444a0a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.318+0000 7f3444a0a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.318+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400abb20 con 0x7f34400aecc0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f343001b3b0 con 0x7f34400aecc0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f342e7fc700 1 --2- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 0x7f342006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f343e59c700 1 --2- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 0x7f342006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3430083350 con 0x7f34400aecc0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f343e59c700 10 monclient: get_auth_request con 0x7f3420068890 auth_method 0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f342e7fc700 10 client.4479.objecter ms_dispatch 0x7f3440001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.319+0000 7f342e7fc700 3 client.4479.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.320+0000 7f342e7fc700 3 client.4479.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:20.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.320+0000 7f342e7fc700 20 client.4479.objecter dump_active .. 0 homeless 2023-09-30T01:04:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.320+0000 7f342e7fc700 10 client.4479.objecter ms_handle_connect 0x7f3420068890 2023-09-30T01:04:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.320+0000 7f343e59c700 1 --2- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 0x7f342006ad40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34340091e0 tx=0x7f3434017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.324+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3430050c10 con 0x7f34400aecc0 2023-09-30T01:04:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.324+0000 7f342e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:20.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.324+0000 7f342e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.545+0000 7f3444a0a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.545+0000 7f3444a0a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.545+0000 7f3444a0a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.545+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3440072960 con 0x7f34400aecc0 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.546+0000 7f342e7fc700 1 -- 172.21.15.115:0/4289560675 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3430054230 con 0x7f34400aecc0 2023-09-30T01:04:20.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.546+0000 7f342e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:20.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.546+0000 7f342e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:20.547 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:20.547 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:20.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.547+0000 7f3444a0a700 10 librados: watch_flush enter 2023-09-30T01:04:20.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.547+0000 7f3444a0a700 10 librados: watch_flush exit 2023-09-30T01:04:20.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 20 client.4479.objecter shutdown clearing up homeless session... 2023-09-30T01:04:20.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 10 client.4479.objecter successfully canceled tick 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 msgr2=0x7f342006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 0x7f342006ad40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f34340091e0 tx=0x7f3434017000 comp rx=0 tx=0).stop 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 10 monclient: shutdown 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 msgr2=0x7f3440151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f34300011e0 tx=0x7f3430007d30 comp rx=0 tx=0).stop 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 shutdown_connections 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> v2:172.21.15.115:6816/115635 conn(0x7f3420068890 0x7f342006ad40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34401415c0 0x7f34400abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34400ae8f0 0x7f344013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 --2- 172.21.15.115:0/4289560675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34400aecc0 0x7f3440151460 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:20.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 >> 172.21.15.115:0/4289560675 conn(0x7f344006a460 msgr2=0x7f34400b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:20.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 shutdown_connections 2023-09-30T01:04:20.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 -- 172.21.15.115:0/4289560675 wait complete. 2023-09-30T01:04:20.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:20.548+0000 7f3444a0a700 1 librados: shutdown 2023-09-30T01:04:23.560 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 10 monclient: get_monmap_and_config 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 10 monclient: build_initial_monmap 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 10 monclient: monmap: 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:23.689087+0000 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:23.689087+0000 2023-09-30T01:04:23.688 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:23.689 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 1 Processor -- start 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 1 -- start start 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.688+0000 7f854306a700 10 monclient: init 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f853c0ae8f0 0x7f853c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): picked mon.noname-a con 0x7f853c0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f853c0aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f853c1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0a7040 con 0x7f853c0aecc0 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0a57f0 con 0x7f853c1415c0 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c0a5480 con 0x7f853c0ae8f0 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f854306a700 10 monclient(hunting): authenticate will time out at 1125.139502s 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f853c0ae8f0 0x7f853c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f853bfff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f853c0aecc0 0x7f853c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56668/0 (socket says 172.21.15.115:56668) 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f853bfff700 1 -- 172.21.15.115:0/1414473095 learned_addr learned my addr 172.21.15.115:0/1414473095 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): get_auth_request con 0x7f853c0ae8f0 auth_method 0 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient(hunting): handle_auth_done global_id 4482 payload 274 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 1 -- 172.21.15.115:0/1414473095 >> v1:172.21.15.157:6789/0 conn(0x7f853c1415c0 legacy=0x7f853c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 1 -- 172.21.15.115:0/1414473095 >> v1:172.21.15.115:6789/0 conn(0x7f853c0aecc0 legacy=0x7f853c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient: _finish_hunting 0 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 1 monclient: found mon.noname-a 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f853b7fe700 1 -- 172.21.15.115:0/1414473095 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062867739 0 0) 0x7f853c0a7040 con 0x7f853c0aecc0 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f8540e06700 1 -- 172.21.15.115:0/1414473095 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0a55c0 con 0x7f853c0ae8f0 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.689+0000 7f853b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f8540e06700 1 --2- 172.21.15.115:0/1414473095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a5d10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f85300098e0 tx=0x7f85300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d98cdda5f840953 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 1 -- 172.21.15.115:0/1414473095 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530009e80 con 0x7f853c0ae8f0 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: dump: 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.696 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 5 monclient: authenticate success, global_id 4482 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 1 -- 172.21.15.115:0/1414473095 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853000b820 con 0x7f853c0ae8f0 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:23.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 4 set_mon_vals no callback set 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 1 -- 172.21.15.115:0/1414473095 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000bba0 con 0x7f853c0ae8f0 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f853b7fe700 10 monclient: dump: 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.698 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 10 monclient: shutdown 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 -- 172.21.15.115:0/1414473095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 msgr2=0x7f853c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 --2- 172.21.15.115:0/1414473095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a5d10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f85300098e0 tx=0x7f85300091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 -- 172.21.15.115:0/1414473095 shutdown_connections 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 --2- 172.21.15.115:0/1414473095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a5d10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 -- 172.21.15.115:0/1414473095 >> 172.21.15.115:0/1414473095 conn(0x7f853c06a460 msgr2=0x7f853c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:23.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 -- 172.21.15.115:0/1414473095 shutdown_connections 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.690+0000 7f854306a700 1 -- 172.21.15.115:0/1414473095 wait complete. 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 10 monclient: build_initial_monmap 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 10 monclient: monmap: 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:23.706 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 librados: starting msgr at 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 librados: starting objecter 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 Processor -- start 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 -- start start 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 librados: setting wanted keys 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 1 librados: calling monclient init 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.691+0000 7f854306a700 10 monclient: init 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:23.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): picked mon.noname-a con 0x7f853c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f853c0aecc0 0x7f853c13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): picked mon.noname-b con 0x7f853c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f853c1415c0 0x7f853c13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): picked mon.noname-c con 0x7f853c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c0a8f50 con 0x7f853c0ae8f0 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c0aa230 con 0x7f853c0aecc0 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:23.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c0aa4a0 con 0x7f853c1415c0 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f854306a700 10 monclient(hunting): authenticate will time out at 1125.142500s 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:35738/0 (socket says 172.21.15.115:35738) 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 -- 172.21.15.115:0/3042694153 learned_addr learned my addr 172.21.15.115:0/3042694153 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): get_auth_request con 0x7f853c0ae8f0 auth_method 0 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f853c1415c0 0x7f853c13d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:23.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): get_auth_request con 0x7f853c1415c0 auth_method 0 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8541607700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f853bfff700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f853c0aecc0 0x7f853c13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:23.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f853bfff700 10 monclient(hunting): get_auth_request con 0x7f853c0aecc0 auth_method 0 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f853bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f853bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f853bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient(hunting): handle_auth_done global_id 4488 payload 824 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 -- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f853c1415c0 msgr2=0x7f853c13d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f853c1415c0 0x7f853c13d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 -- 172.21.15.115:0/3042694153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f853c0aecc0 msgr2=0x7f853c13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f853c0aecc0 0x7f853c13d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient: _finish_hunting 0 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 monclient: found mon.noname-a 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:23.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.692+0000 7f8540e06700 1 -- 172.21.15.115:0/3042694153 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530009420 con 0x7f853c0ae8f0 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8540e06700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f85300030c0 tx=0x7f8530016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 client.?.objecter ms_handle_connect 0x7f853c0ae8f0 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000b990 con 0x7f853c0ae8f0 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: dump: 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:23.712 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: _finish_auth 0 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530020070 con 0x7f853c0ae8f0 2023-09-30T01:04:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 5 monclient: authenticate success, global_id 4488 2023-09-30T01:04:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 10 monclient: _renew_subs 2023-09-30T01:04:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c0aa780 con 0x7f853c0ae8f0 2023-09-30T01:04:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853001b450 con 0x7f853c0ae8f0 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f8538ff9700 10 monclient: dump: 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f853affd700 4 set_mon_vals no callback set 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 10 monclient: _renew_subs 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c0a8ae0 con 0x7f853c0ae8f0 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.693+0000 7f854306a700 1 librados: init done 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f854306a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f854306a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f854306a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c0aa910 con 0x7f853c0ae8f0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8530022030 con 0x7f853c0ae8f0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.694+0000 7f8538ff9700 1 --2- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 0x7f852006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8530085ee0 con 0x7f853c0ae8f0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 10 client.4488.objecter ms_dispatch 0x7f853c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 3 client.4488.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 3 client.4488.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f853bfff700 1 --2- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 0x7f852006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f853bfff700 10 monclient: get_auth_request con 0x7f8520068890 auth_method 0 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 20 client.4488.objecter dump_active .. 0 homeless 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f8538ff9700 10 client.4488.objecter ms_handle_connect 0x7f8520068890 2023-09-30T01:04:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.695+0000 7f853bfff700 1 --2- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 0x7f852006ad40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f853c141e20 tx=0x7f8528009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.699+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8530053700 con 0x7f853c0ae8f0 2023-09-30T01:04:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.699+0000 7f8538ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.699+0000 7f8538ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.915+0000 7f854306a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.915+0000 7f854306a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.915+0000 7f854306a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.915+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f853c0aa910 con 0x7f853c0ae8f0 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.916+0000 7f8538ff9700 1 -- 172.21.15.115:0/3042694153 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8530056d20 con 0x7f853c0ae8f0 2023-09-30T01:04:23.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.916+0000 7f8538ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:23.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.916+0000 7f8538ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:23.917 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:23.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.917+0000 7f854306a700 10 librados: watch_flush enter 2023-09-30T01:04:23.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 10 librados: watch_flush exit 2023-09-30T01:04:23.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 20 client.4488.objecter shutdown clearing up homeless session... 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 10 client.4488.objecter successfully canceled tick 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 msgr2=0x7f852006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 0x7f852006ad40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f853c141e20 tx=0x7f8528009040 comp rx=0 tx=0).stop 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 10 monclient: shutdown 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 msgr2=0x7f853c0a82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f85300030c0 tx=0x7f8530016000 comp rx=0 tx=0).stop 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 shutdown_connections 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> v2:172.21.15.115:6816/115635 conn(0x7f8520068890 0x7f852006ad40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f853c1415c0 0x7f853c13d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f853c0ae8f0 0x7f853c0a82e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 --2- 172.21.15.115:0/3042694153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f853c0aecc0 0x7f853c13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:23.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 >> 172.21.15.115:0/3042694153 conn(0x7f853c06a460 msgr2=0x7f853c13ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:23.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 shutdown_connections 2023-09-30T01:04:23.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 -- 172.21.15.115:0/3042694153 wait complete. 2023-09-30T01:04:23.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:23.918+0000 7f854306a700 1 librados: shutdown 2023-09-30T01:04:26.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:27.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.058+0000 7f21cb650700 10 monclient: get_monmap_and_config 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.058+0000 7f21cb650700 10 monclient: build_initial_monmap 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.058+0000 7f21cb650700 10 monclient: monmap: 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:27.059768+0000 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:27.059768+0000 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.059 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:27.060 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:27.060 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:27.060 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:27.060 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:27.060 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.059+0000 7f21cb650700 1 Processor -- start 2023-09-30T01:04:27.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.059+0000 7f21cb650700 1 -- start start 2023-09-30T01:04:27.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.059+0000 7f21cb650700 10 monclient: init 2023-09-30T01:04:27.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f21c40ae8f0 0x7f21c40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-a con 0x7f21c40ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21c40aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 0x7f21c40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-b con 0x7f21c41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c40a7040 con 0x7f21c40aecc0 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c40a57f0 con 0x7f21c40ae8f0 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40a5480 con 0x7f21c41415c0 2023-09-30T01:04:27.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21cb650700 10 monclient(hunting): authenticate will time out at 1128.510312s 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 0x7f21c40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f21c40ae8f0 0x7f21c40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c8beb700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f21c40aecc0 0x7f21c40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56682/0 (socket says 172.21.15.115:56682) 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c8beb700 1 -- 172.21.15.115:0/2550869019 learned_addr learned my addr 172.21.15.115:0/2550869019 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): get_auth_request con 0x7f21c41415c0 auth_method 0 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): get_auth_request con 0x7f21c40ae8f0 auth_method 0 2023-09-30T01:04:27.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient(hunting): handle_auth_done global_id 4491 payload 274 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 -- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 msgr2=0x7f21c40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 0x7f21c40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 -- 172.21.15.115:0/2550869019 >> v1:172.21.15.115:6789/0 conn(0x7f21c40aecc0 legacy=0x7f21c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient: _finish_hunting 0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 monclient: found mon.noname-a 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 -- 172.21.15.115:0/2550869019 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4141a40 con 0x7f21c40ae8f0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21bbfff700 1 -- 172.21.15.115:0/2550869019 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192220859 0 0) 0x7f21c40a7040 con 0x7f21c40aecc0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21bbfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c9bed700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 0x7f21c40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.060+0000 7f21c93ec700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c40ae8f0 0x7f21c40a5d10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f21b40098e0 tx=0x7f21b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74d8e5b0c4776b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 1 -- 172.21.15.115:0/2550869019 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400b640 con 0x7f21c40ae8f0 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:27.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: dump: 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: _finish_auth 0 2023-09-30T01:04:27.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 5 monclient: authenticate success, global_id 4491 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 1 -- 172.21.15.115:0/2550869019 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b400b7a0 con 0x7f21c40ae8f0 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 4 set_mon_vals no callback set 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 1 -- 172.21.15.115:0/2550869019 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400bb40 con 0x7f21c40ae8f0 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21bbfff700 10 monclient: dump: 2023-09-30T01:04:27.074 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 10 monclient: shutdown 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:27.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 -- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c40ae8f0 msgr2=0x7f21c40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c40ae8f0 0x7f21c40a5d10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f21b40098e0 tx=0x7f21b40091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 -- 172.21.15.115:0/2550869019 shutdown_connections 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c41415c0 0x7f21c40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 --2- 172.21.15.115:0/2550869019 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c40ae8f0 0x7f21c40a5d10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 -- 172.21.15.115:0/2550869019 >> 172.21.15.115:0/2550869019 conn(0x7f21c406a460 msgr2=0x7f21c40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 -- 172.21.15.115:0/2550869019 shutdown_connections 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.061+0000 7f21cb650700 1 -- 172.21.15.115:0/2550869019 wait complete. 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 10 monclient: build_initial_monmap 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 10 monclient: monmap: 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:27.076 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 librados: starting msgr at 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 librados: starting objecter 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 Processor -- start 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 -- start start 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 librados: setting wanted keys 2023-09-30T01:04:27.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 1 librados: calling monclient init 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.062+0000 7f21cb650700 10 monclient: init 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-c con 0x7f21c40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21c40aecc0 0x7f21c406b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-b con 0x7f21c40aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 0x7f21c406c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): picked mon.noname-a con 0x7f21c41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c413e850 con 0x7f21c41415c0 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40ad8d0 con 0x7f21c40aecc0 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c406ac50 con 0x7f21c40ae8f0 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21cb650700 10 monclient(hunting): authenticate will time out at 1128.513594s 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 0x7f21c406c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48088/0 (socket says 172.21.15.115:48088) 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 1 -- 172.21.15.115:0/1004190613 learned_addr learned my addr 172.21.15.115:0/1004190613 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): get_auth_request con 0x7f21c40ae8f0 auth_method 0 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): get_auth_request con 0x7f21c41415c0 auth_method 0 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:27.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c93ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c8beb700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21c40aecc0 0x7f21c406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 1 -- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 msgr2=0x7f21c406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.063+0000 7f21c9bed700 1 -- 172.21.15.115:0/1004190613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21c40aecc0 msgr2=0x7f21c406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21c40aecc0 0x7f21c406b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 10 monclient: _finish_hunting 0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 1 monclient: found mon.noname-a 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 1 -- 172.21.15.115:0/1004190613 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc009a20 con 0x7f21c41415c0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c93ec700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21c9bed700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 0x7f21c406c460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e70 tx=0x7f21bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 client.?.objecter ms_handle_connect 0x7f21c41415c0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc00cb50 con 0x7f21c41415c0 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:27.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: dump: 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: _finish_auth 0 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:27.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc00f040 con 0x7f21c41415c0 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 5 monclient: authenticate success, global_id 4497 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc014480 con 0x7f21c41415c0 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 10 monclient: _renew_subs 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b4009420 con 0x7f21c41415c0 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21bb7fe700 4 set_mon_vals no callback set 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21b97fa700 10 monclient: dump: 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:27.084 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 10 monclient: _renew_subs 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c406c9a0 con 0x7f21c41415c0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.064+0000 7f21cb650700 1 librados: init done 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.065+0000 7f21cb650700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.065+0000 7f21cb650700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.065+0000 7f21cb650700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.065+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c40b1870 con 0x7f21c41415c0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.065+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f21bc003020 con 0x7f21c41415c0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 1 --2- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 0x7f21a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21c93ec700 1 --2- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 0x7f21a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f21bc084730 con 0x7f21c41415c0 2023-09-30T01:04:27.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 10 client.4497.objecter ms_dispatch 0x7f21c4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21c93ec700 10 monclient: get_auth_request con 0x7f21a8068890 auth_method 0 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 3 client.4497.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 3 client.4497.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 20 client.4497.objecter dump_active .. 0 homeless 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21b97fa700 10 client.4497.objecter ms_handle_connect 0x7f21a8068890 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.066+0000 7f21c93ec700 1 --2- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 0x7f21a806ad40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21b4003080 tx=0x7f21b4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.070+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21bc051ff0 con 0x7f21c41415c0 2023-09-30T01:04:27.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.070+0000 7f21b97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:27.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.070+0000 7f21b97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.291+0000 7f21cb650700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.291+0000 7f21cb650700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.291+0000 7f21cb650700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.291+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f21c4072960 con 0x7f21c41415c0 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.292+0000 7f21b97fa700 1 -- 172.21.15.115:0/1004190613 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f21bc055610 con 0x7f21c41415c0 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.292+0000 7f21b97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:27.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.292+0000 7f21b97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:27.293 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:27.293 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 10 librados: watch_flush enter 2023-09-30T01:04:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 10 librados: watch_flush exit 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 20 client.4497.objecter shutdown clearing up homeless session... 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 10 client.4497.objecter successfully canceled tick 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 msgr2=0x7f21a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 0x7f21a806ad40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21b4003080 tx=0x7f21b4017000 comp rx=0 tx=0).stop 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 10 monclient: shutdown 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 msgr2=0x7f21c406c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 0x7f21c406c460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e70 tx=0x7f21bc00b040 comp rx=0 tx=0).stop 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 shutdown_connections 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> v2:172.21.15.115:6816/115635 conn(0x7f21a8068890 0x7f21a806ad40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21c41415c0 0x7f21c406c460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f21c40ae8f0 0x7f21c406e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 --2- 172.21.15.115:0/1004190613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21c40aecc0 0x7f21c406b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 >> 172.21.15.115:0/1004190613 conn(0x7f21c406a460 msgr2=0x7f21c413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 shutdown_connections 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 -- 172.21.15.115:0/1004190613 wait complete. 2023-09-30T01:04:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:27.294+0000 7f21cb650700 1 librados: shutdown 2023-09-30T01:04:30.306 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:30.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.437+0000 7f41927c9700 10 monclient: get_monmap_and_config 2023-09-30T01:04:30.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.437+0000 7f41927c9700 10 monclient: build_initial_monmap 2023-09-30T01:04:30.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.437+0000 7f41927c9700 10 monclient: monmap: 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:30.438397+0000 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:30.438397+0000 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:30.438 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:30.439 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 Processor -- start 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 -- start start 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient: init 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 0x7f418c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-a con 0x7f418c0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-b con 0x7f418c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f418c1415c0 0x7f418c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-c con 0x7f418c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0a7040 con 0x7f418c0ae8f0 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0b19b0 con 0x7f418c1415c0 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0a57f0 con 0x7f418c0aecc0 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f41927c9700 10 monclient(hunting): authenticate will time out at 1131.888836s 2023-09-30T01:04:30.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f418bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 0x7f418c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48090/0 (socket says 172.21.15.115:48090) 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.438+0000 7f418b7fe700 1 -- 172.21.15.115:0/4189666689 learned_addr learned my addr 172.21.15.115:0/4189666689 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): get_auth_request con 0x7f418c0aecc0 auth_method 0 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): get_auth_request con 0x7f418c0ae8f0 auth_method 0 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f4190d66700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.157:3300/0 conn(0x7f418c1415c0 0x7f418c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f4190d66700 10 monclient(hunting): get_auth_request con 0x7f418c1415c0 auth_method 0 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f4190d66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f4190d66700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f4190d66700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2023-09-30T01:04:30.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 -- 172.21.15.115:0/4189666689 >> v2:172.21.15.157:3300/0 conn(0x7f418c1415c0 msgr2=0x7f418c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.157:3300/0 conn(0x7f418c1415c0 0x7f418c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 -- 172.21.15.115:0/4189666689 >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 msgr2=0x7f418c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 0x7f418c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 -- 172.21.15.115:0/4189666689 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0a5480 con 0x7f418c0aecc0 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418bfff700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 0x7f418c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418b7fe700 1 --2- 172.21.15.115:0/4189666689 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f418c0b0340 tx=0x7f4180009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628628d5a1aebb85 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 1 -- 172.21.15.115:0/4189666689 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180009e70 con 0x7f418c0aecc0 2023-09-30T01:04:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: dump: 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:30.453 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: _finish_auth 0 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 1 -- 172.21.15.115:0/4189666689 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000b860 con 0x7f418c0aecc0 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f41927c9700 5 monclient: authenticate success, global_id 4376 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f41927c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f41927c9700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 1 -- 172.21.15.115:0/4189666689 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418000bc00 con 0x7f418c0aecc0 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f41927c9700 4 set_mon_vals no callback set 2023-09-30T01:04:30.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.439+0000 7f418affd700 10 monclient: dump: 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 10 monclient: shutdown 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 -- 172.21.15.115:0/4189666689 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 msgr2=0x7f418c0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 --2- 172.21.15.115:0/4189666689 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f418c0b0340 tx=0x7f4180009250 comp rx=0 tx=0).stop 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 -- 172.21.15.115:0/4189666689 shutdown_connections 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.115:3300/0 conn(0x7f418c0ae8f0 0x7f418c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 --2- 172.21.15.115:0/4189666689 >> v2:172.21.15.157:3300/0 conn(0x7f418c1415c0 0x7f418c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 --2- 172.21.15.115:0/4189666689 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c0b1470 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 -- 172.21.15.115:0/4189666689 >> 172.21.15.115:0/4189666689 conn(0x7f418c06a460 msgr2=0x7f418c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 -- 172.21.15.115:0/4189666689 shutdown_connections 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 1 -- 172.21.15.115:0/4189666689 wait complete. 2023-09-30T01:04:30.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 10 monclient: build_initial_monmap 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.440+0000 7f41927c9700 10 monclient: monmap: 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 librados: starting msgr at 2023-09-30T01:04:30.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 librados: starting objecter 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 Processor -- start 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 -- start start 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 librados: setting wanted keys 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 librados: calling monclient init 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient: init 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 0x7f418c13e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-a con 0x7f418c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-c con 0x7f418c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:30.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 0x7f418c0a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): picked mon.noname-b con 0x7f418c1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c14e4d0 con 0x7f418c0ae8f0 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c13ef50 con 0x7f418c1415c0 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c14e180 con 0x7f418c0aecc0 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.441+0000 7f41927c9700 10 monclient(hunting): authenticate will time out at 1131.891944s 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 0x7f418c13e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48096/0 (socket says 172.21.15.115:48096) 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 -- 172.21.15.115:0/2356999117 learned_addr learned my addr 172.21.15.115:0/2356999117 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): get_auth_request con 0x7f418c0aecc0 auth_method 0 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): get_auth_request con 0x7f418c0ae8f0 auth_method 0 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 0x7f418c0a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 10 monclient(hunting): get_auth_request con 0x7f418c1415c0 auth_method 0 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2023-09-30T01:04:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 -- 172.21.15.115:0/2356999117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 msgr2=0x7f418c0a76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 0x7f418c0a76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 -- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 msgr2=0x7f418c13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 0x7f418c13e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 monclient: found mon.noname-c 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 -- 172.21.15.115:0/2356999117 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009420 con 0x7f418c0aecc0 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4190d66700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 0x7f418c0a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418bfff700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 0x7f418c13e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418b7fe700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f418c0a5930 tx=0x7f4180017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:30.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 client.?.objecter ms_handle_connect 0x7f418c0aecc0 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180000d10 con 0x7f418c0aecc0 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: dump: 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.463 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: _finish_auth 0 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 5 monclient: authenticate success, global_id 4379 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000bbe0 con 0x7f418c0aecc0 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 10 monclient: _renew_subs 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41800094f0 con 0x7f418c0aecc0 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:30.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180011670 con 0x7f418c0aecc0 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f4173fff700 10 monclient: dump: 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:30.465 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f418a7fc700 4 set_mon_vals no callback set 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 10 monclient: _renew_subs 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0a7e20 con 0x7f418c0aecc0 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.442+0000 7f41927c9700 1 librados: init done 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.443+0000 7f41927c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.443+0000 7f41927c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.443+0000 7f41927c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.443+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c0a74f0 con 0x7f418c0aecc0 2023-09-30T01:04:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4180011aa0 con 0x7f418c0aecc0 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 1 --2- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 0x7f417806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4180085410 con 0x7f418c0aecc0 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 10 client.4379.objecter ms_dispatch 0x7f418c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 3 client.4379.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 3 client.4379.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f418bfff700 1 --2- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 0x7f417806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f418bfff700 10 monclient: get_auth_request con 0x7f4178068890 auth_method 0 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 20 client.4379.objecter dump_active .. 0 homeless 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f4173fff700 10 client.4379.objecter ms_handle_connect 0x7f4178068890 2023-09-30T01:04:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.444+0000 7f418bfff700 1 --2- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 0x7f417806ad40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f417c011600 tx=0x7f417c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.448+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4180052cd0 con 0x7f418c0aecc0 2023-09-30T01:04:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.448+0000 7f4173fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.448+0000 7f4173fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:30.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f41927c9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:30.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f41927c9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:30.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f41927c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:30.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f418c072960 con 0x7f418c0aecc0 2023-09-30T01:04:30.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f4173fff700 1 -- 172.21.15.115:0/2356999117 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f41800562f0 con 0x7f418c0aecc0 2023-09-30T01:04:30.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f4173fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:30.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.671+0000 7f4173fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:30.673 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:30.673 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 10 librados: watch_flush enter 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 10 librados: watch_flush exit 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 20 client.4379.objecter shutdown clearing up homeless session... 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 10 client.4379.objecter successfully canceled tick 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 msgr2=0x7f417806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:30.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 0x7f417806ad40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f417c011600 tx=0x7f417c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 10 monclient: shutdown 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 msgr2=0x7f418c14dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f418c0a5930 tx=0x7f4180017000 comp rx=0 tx=0).stop 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 shutdown_connections 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> v2:172.21.15.115:6816/115635 conn(0x7f4178068890 0x7f417806ad40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f418c1415c0 0x7f418c0a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f418c0ae8f0 0x7f418c13e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 --2- 172.21.15.115:0/2356999117 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f418c0aecc0 0x7f418c14dc40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:30.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 >> 172.21.15.115:0/2356999117 conn(0x7f418c06a460 msgr2=0x7f418c142c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:30.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 shutdown_connections 2023-09-30T01:04:30.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 -- 172.21.15.115:0/2356999117 wait complete. 2023-09-30T01:04:30.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:30.673+0000 7f41927c9700 1 librados: shutdown 2023-09-30T01:04:33.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.814+0000 7fcd44d6b700 10 monclient: get_monmap_and_config 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.814+0000 7fcd44d6b700 10 monclient: build_initial_monmap 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.815+0000 7fcd44d6b700 10 monclient: monmap: 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:33.815970+0000 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:33.815970+0000 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:33.815 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:33.816 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.815+0000 7fcd44d6b700 1 Processor -- start 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.815+0000 7fcd44d6b700 1 -- start start 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.815+0000 7fcd44d6b700 10 monclient: init 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd400ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcd400aecc0 0x7fcd400a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-a con 0x7fcd400aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:33.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcd401415c0 0x7fcd400adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-c con 0x7fcd401415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400a7040 con 0x7fcd400ae8f0 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd400a57f0 con 0x7fcd400aecc0 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd400a5480 con 0x7fcd401415c0 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd44d6b700 10 monclient(hunting): authenticate will time out at 1135.266480s 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcd400aecc0 0x7fcd400a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcd400aecc0 0x7fcd400a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:35976/0 (socket says 172.21.15.115:35976) 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 -- 172.21.15.115:0/813380981 learned_addr learned my addr 172.21.15.115:0/813380981 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): get_auth_request con 0x7fcd400aecc0 auth_method 0 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 1 --2- 172.21.15.115:0/813380981 >> v2:172.21.15.157:3300/0 conn(0x7fcd401415c0 0x7fcd400adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): get_auth_request con 0x7fcd401415c0 auth_method 0 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:33.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_done global_id 4506 payload 274 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 -- 172.21.15.115:0/813380981 >> v2:172.21.15.157:3300/0 conn(0x7fcd401415c0 msgr2=0x7fcd400adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 --2- 172.21.15.115:0/813380981 >> v2:172.21.15.157:3300/0 conn(0x7fcd401415c0 0x7fcd400adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 -- 172.21.15.115:0/813380981 >> v1:172.21.15.157:6789/0 conn(0x7fcd400ae8f0 legacy=0x7fcd400b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 monclient: found mon.noname-a 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:33.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.816+0000 7fcd3dd9b700 1 -- 172.21.15.115:0/813380981 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400a55c0 con 0x7fcd400aecc0 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3dd9b700 1 --2- 172.21.15.115:0/813380981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd400aecc0 0x7fcd400a5d10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a6250 tx=0x7fcd30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715edc3f5021e564 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 1 -- 172.21.15.115:0/813380981 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30009e70 con 0x7fcd400aecc0 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: dump: 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:33.828 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.829 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.829 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:33.829 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.829 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:33.829 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: _finish_auth 0 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 1 -- 172.21.15.115:0/813380981 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3000b860 con 0x7fcd400aecc0 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 5 monclient: authenticate success, global_id 4506 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 1 -- 172.21.15.115:0/813380981 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000bc00 con 0x7fcd400aecc0 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 4 set_mon_vals no callback set 2023-09-30T01:04:33.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd3d59a700 10 monclient: dump: 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 10 monclient: shutdown 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 -- 172.21.15.115:0/813380981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd400aecc0 msgr2=0x7fcd400a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/813380981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd400aecc0 0x7fcd400a5d10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a6250 tx=0x7fcd30009250 comp rx=0 tx=0).stop 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 -- 172.21.15.115:0/813380981 shutdown_connections 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/813380981 >> v2:172.21.15.157:3300/0 conn(0x7fcd401415c0 0x7fcd400adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/813380981 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd400aecc0 0x7fcd400a5d10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 -- 172.21.15.115:0/813380981 >> 172.21.15.115:0/813380981 conn(0x7fcd4006a460 msgr2=0x7fcd400630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 -- 172.21.15.115:0/813380981 shutdown_connections 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.817+0000 7fcd44d6b700 1 -- 172.21.15.115:0/813380981 wait complete. 2023-09-30T01:04:33.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 10 monclient: build_initial_monmap 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 10 monclient: monmap: 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 librados: starting msgr at 2023-09-30T01:04:33.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 librados: starting objecter 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 Processor -- start 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 -- start start 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 librados: setting wanted keys 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 1 librados: calling monclient init 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.818+0000 7fcd44d6b700 10 monclient: init 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd400ae8f0 0x7fcd4013e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-b con 0x7fcd400ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-c con 0x7fcd400aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:33.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): picked mon.noname-a con 0x7fcd401415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400aa410 con 0x7fcd401415c0 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400ab7b0 con 0x7fcd400ae8f0 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40151ca0 con 0x7fcd400aecc0 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd44d6b700 10 monclient(hunting): authenticate will time out at 1135.269442s 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:35986/0 (socket says 172.21.15.115:35986) 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 -- 172.21.15.115:0/1456429567 learned_addr learned my addr 172.21.15.115:0/1456429567 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): get_auth_request con 0x7fcd401415c0 auth_method 0 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38040/0 (socket says 172.21.15.115:38040) 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): get_auth_request con 0x7fcd400aecc0 auth_method 0 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:33.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3e59c700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd400ae8f0 0x7fcd4013e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3e59c700 10 monclient(hunting): get_auth_request con 0x7fcd400ae8f0 auth_method 0 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 -- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 msgr2=0x7fcd40151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 -- 172.21.15.115:0/1456429567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd400ae8f0 msgr2=0x7fcd4013e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd400ae8f0 0x7fcd4013e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 monclient: found mon.noname-a 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3ed9d700 1 -- 172.21.15.115:0/1456429567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd300094f0 con 0x7fcd401415c0 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.819+0000 7fcd3dd9b700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd3ed9d700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e20 tx=0x7fcd3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 client.?.objecter ms_handle_connect 0x7fcd401415c0 2023-09-30T01:04:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3800cb50 con 0x7fcd401415c0 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: dump: 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3800f040 con 0x7fcd401415c0 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 5 monclient: authenticate success, global_id 4509 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 10 monclient: _renew_subs 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd400aa900 con 0x7fcd401415c0 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38014480 con 0x7fcd401415c0 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd2e7fc700 10 monclient: dump: 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd3cd99700 4 set_mon_vals no callback set 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 10 monclient: _renew_subs 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd400aa110 con 0x7fcd401415c0 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.820+0000 7fcd44d6b700 1 librados: init done 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.821+0000 7fcd44d6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.821+0000 7fcd44d6b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.821+0000 7fcd44d6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.821+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd400ab980 con 0x7fcd401415c0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.821+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcd38014830 con 0x7fcd401415c0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 1 --2- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 0x7fcd2006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcd38084f80 con 0x7fcd401415c0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 10 client.4509.objecter ms_dispatch 0x7fcd40001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 3 client.4509.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 3 client.4509.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd3e59c700 1 --2- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 0x7fcd2006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd3e59c700 10 monclient: get_auth_request con 0x7fcd20068890 auth_method 0 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 20 client.4509.objecter dump_active .. 0 homeless 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd2e7fc700 10 client.4509.objecter ms_handle_connect 0x7fcd20068890 2023-09-30T01:04:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.822+0000 7fcd3e59c700 1 --2- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 0x7fcd2006ad40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd34003860 tx=0x7fcd34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:33.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.826+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd380528c0 con 0x7fcd401415c0 2023-09-30T01:04:33.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.826+0000 7fcd2e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:33.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:33.826+0000 7fcd2e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:34.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd44d6b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:34.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd44d6b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd44d6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd40072960 con 0x7fcd401415c0 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd2e7fc700 1 -- 172.21.15.115:0/1456429567 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcd38055ee0 con 0x7fcd401415c0 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd2e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.046+0000 7fcd2e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:34.047 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 10 librados: watch_flush enter 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 10 librados: watch_flush exit 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 20 client.4509.objecter shutdown clearing up homeless session... 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 10 client.4509.objecter successfully canceled tick 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 msgr2=0x7fcd2006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 0x7fcd2006ad40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd34003860 tx=0x7fcd34009040 comp rx=0 tx=0).stop 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 10 monclient: shutdown 2023-09-30T01:04:34.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 msgr2=0x7fcd40151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e20 tx=0x7fcd3800b040 comp rx=0 tx=0).stop 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 shutdown_connections 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> v2:172.21.15.115:6816/115635 conn(0x7fcd20068890 0x7fcd2006ad40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcd401415c0 0x7fcd40151760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd400ae8f0 0x7fcd4013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 --2- 172.21.15.115:0/1456429567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcd400aecc0 0x7fcd40151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 >> 172.21.15.115:0/1456429567 conn(0x7fcd4006a460 msgr2=0x7fcd400b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 shutdown_connections 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 -- 172.21.15.115:0/1456429567 wait complete. 2023-09-30T01:04:34.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:34.048+0000 7fcd44d6b700 1 librados: shutdown 2023-09-30T01:04:37.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:37.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.189+0000 7f79b145b700 10 monclient: get_monmap_and_config 2023-09-30T01:04:37.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.189+0000 7f79b145b700 10 monclient: build_initial_monmap 2023-09-30T01:04:37.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.189+0000 7f79b145b700 10 monclient: monmap: 2023-09-30T01:04:37.190 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:37.190780+0000 2023-09-30T01:04:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:37.190780+0000 2023-09-30T01:04:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.190+0000 7f79b145b700 1 Processor -- start 2023-09-30T01:04:37.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.190+0000 7f79b145b700 1 -- start start 2023-09-30T01:04:37.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.190+0000 7f79b145b700 10 monclient: init 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.190+0000 7f79b145b700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79ac0ac750 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79ac0aecf0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-b con 0x7f79ac0acb20 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac06ef30 con 0x7f79ac0aecf0 2023-09-30T01:04:37.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac139b20 con 0x7f79ac0ac750 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac06d3e0 con 0x7f79ac0acb20 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient(hunting): authenticate will time out at 1138.641198s 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79aa7fc700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f79ac0aecf0 0x7f79ac13bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:38404/0 (socket says 172.21.15.115:38404) 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79aa7fc700 1 -- 172.21.15.115:0/2503706397 learned_addr learned my addr 172.21.15.115:0/2503706397 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): get_auth_request con 0x7f79ac0acb20 auth_method 0 2023-09-30T01:04:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 1 -- 172.21.15.115:0/2503706397 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830540054 0 0) 0x7f79ac06ef30 con 0x7f79ac0aecf0 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 -- 172.21.15.115:0/2503706397 >> v1:172.21.15.157:6789/0 conn(0x7f79ac0ac750 legacy=0x7f79ac06dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 -- 172.21.15.115:0/2503706397 >> v1:172.21.15.115:6789/0 conn(0x7f79ac0aecf0 legacy=0x7f79ac13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:04:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 monclient: found mon.noname-b 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 -- 172.21.15.115:0/2503706397 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac06d680 con 0x7f79ac0acb20 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79ab7fe700 1 --2- 172.21.15.115:0/2503706397 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0abaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009e90 tx=0x7f79a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c46416bffd6d13 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:37.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 1 -- 172.21.15.115:0/2503706397 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400e070 con 0x7f79ac0acb20 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: dump: 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:37.205 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 5 monclient: authenticate success, global_id 4385 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 1 -- 172.21.15.115:0/2503706397 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a400bc30 con 0x7f79ac0acb20 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79b145b700 4 set_mon_vals no callback set 2023-09-30T01:04:37.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 1 -- 172.21.15.115:0/2503706397 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4012430 con 0x7f79ac0acb20 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.191+0000 7f79a9ffb700 10 monclient: dump: 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:37.207 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 10 monclient: shutdown 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 -- 172.21.15.115:0/2503706397 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 msgr2=0x7f79ac0abaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 --2- 172.21.15.115:0/2503706397 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0abaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009e90 tx=0x7f79a4009780 comp rx=0 tx=0).stop 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 -- 172.21.15.115:0/2503706397 shutdown_connections 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 --2- 172.21.15.115:0/2503706397 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0abaf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 -- 172.21.15.115:0/2503706397 >> 172.21.15.115:0/2503706397 conn(0x7f79ac0679e0 msgr2=0x7f79ac0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 -- 172.21.15.115:0/2503706397 shutdown_connections 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.192+0000 7f79b145b700 1 -- 172.21.15.115:0/2503706397 wait complete. 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 10 monclient: build_initial_monmap 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 10 monclient: monmap: 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 librados: starting msgr at 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 librados: starting objecter 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 Processor -- start 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 -- start start 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 librados: setting wanted keys 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 1 librados: calling monclient init 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.193+0000 7f79b145b700 10 monclient: init 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-a con 0x7f79ac0ac750 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-c con 0x7f79ac0acb20 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79ac0aecf0 0x7f79ac0a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): picked mon.noname-b con 0x7f79ac0aecf0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0a9280 con 0x7f79ac0ac750 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0a9490 con 0x7f79ac0aecf0 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0a62a0 con 0x7f79ac0acb20 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient(hunting): authenticate will time out at 1138.644220s 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:36002/0 (socket says 172.21.15.115:36002) 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 -- 172.21.15.115:0/922037658 learned_addr learned my addr 172.21.15.115:0/922037658 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): get_auth_request con 0x7f79ac0ac750 auth_method 0 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:37.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38052/0 (socket says 172.21.15.115:38052) 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): get_auth_request con 0x7f79ac0acb20 auth_method 0 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient(hunting): handle_auth_done global_id 4515 payload 824 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 -- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 msgr2=0x7f79ac0a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 -- 172.21.15.115:0/922037658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79ac0aecf0 msgr2=0x7f79ac0a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79ac0aecf0 0x7f79ac0a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 monclient: found mon.noname-a 2023-09-30T01:04:37.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 -- 172.21.15.115:0/922037658 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a4009a20 con 0x7f79ac0ac750 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aa7fc700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79aaffd700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c7c0 tx=0x7f799c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 client.?.objecter ms_handle_connect 0x7f79ac0ac750 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c011590 con 0x7f79ac0ac750 2023-09-30T01:04:37.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: dump: 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:37.216 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: _finish_auth 0 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 5 monclient: authenticate success, global_id 4515 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c011710 con 0x7f79ac0ac750 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _renew_subs 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac0a5f30 con 0x7f79ac0ac750 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c011ad0 con 0x7f79ac0ac750 2023-09-30T01:04:37.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f7992ffd700 10 monclient: dump: 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:37.218 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _renew_subs 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79a97fa700 4 set_mon_vals no callback set 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac0a77a0 con 0x7f79ac0ac750 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.194+0000 7f79b145b700 1 librados: init done 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.195+0000 7f79b145b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.195+0000 7f79b145b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:37.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.195+0000 7f79b145b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.195+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac072960 con 0x7f79ac0ac750 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f799c01b3b0 con 0x7f79ac0ac750 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 1 --2- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 0x7f799806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f799c0833e0 con 0x7f79ac0ac750 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f79aa7fc700 1 --2- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 0x7f799806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 10 client.4515.objecter ms_dispatch 0x7f79ac001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 3 client.4515.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 3 client.4515.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f79aa7fc700 10 monclient: get_auth_request con 0x7f7998068910 auth_method 0 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.196+0000 7f7992ffd700 20 client.4515.objecter dump_active .. 0 homeless 2023-09-30T01:04:37.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.197+0000 7f7992ffd700 10 client.4515.objecter ms_handle_connect 0x7f7998068910 2023-09-30T01:04:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.197+0000 7f79aa7fc700 1 --2- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 0x7f799806adc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f79ac13f280 tx=0x7f79a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.200+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f799c050cb0 con 0x7f79ac0ac750 2023-09-30T01:04:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.200+0000 7f7992ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:37.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.200+0000 7f7992ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.421+0000 7f79b145b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.421+0000 7f79b145b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.421+0000 7f79b145b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.421+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f79ac0af5e0 con 0x7f79ac0ac750 2023-09-30T01:04:37.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.422+0000 7f7992ffd700 1 -- 172.21.15.115:0/922037658 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f799c017080 con 0x7f79ac0ac750 2023-09-30T01:04:37.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.422+0000 7f7992ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:37.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.422+0000 7f7992ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:37.424 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:37.424 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 10 librados: watch_flush enter 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 10 librados: watch_flush exit 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 20 client.4515.objecter shutdown clearing up homeless session... 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 10 client.4515.objecter successfully canceled tick 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 msgr2=0x7f799806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 0x7f799806adc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f79ac13f280 tx=0x7f79a0009040 comp rx=0 tx=0).stop 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 10 monclient: shutdown 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 msgr2=0x7f79ac13f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c7c0 tx=0x7f799c007f70 comp rx=0 tx=0).stop 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 shutdown_connections 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> v2:172.21.15.115:6816/115635 conn(0x7f7998068910 0x7f799806adc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79ac0aecf0 0x7f79ac0a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79ac0ac750 0x7f79ac13f580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 --2- 172.21.15.115:0/922037658 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79ac0acb20 0x7f79ac0a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:37.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 >> 172.21.15.115:0/922037658 conn(0x7f79ac0679e0 msgr2=0x7f79ac142870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:37.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 shutdown_connections 2023-09-30T01:04:37.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 -- 172.21.15.115:0/922037658 wait complete. 2023-09-30T01:04:37.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:37.424+0000 7f79b145b700 1 librados: shutdown 2023-09-30T01:04:40.436 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.563+0000 7f918a8b4700 10 monclient: get_monmap_and_config 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.563+0000 7f918a8b4700 10 monclient: build_initial_monmap 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.563+0000 7f918a8b4700 10 monclient: monmap: 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:40.564283+0000 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:40.564283+0000 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:40.564 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 Processor -- start 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 -- start start 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient: init 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91840ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f91840aecc0 0x7f91840a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-c con 0x7f91840aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-b con 0x7f91841415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91840a7040 con 0x7f91840ae8f0 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91840a57f0 con 0x7f91840aecc0 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91840a5480 con 0x7f91841415c0 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f918a8b4700 10 monclient(hunting): authenticate will time out at 1142.014775s 2023-09-30T01:04:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38054/0 (socket says 172.21.15.115:38054) 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 1 -- 172.21.15.115:0/1224464909 learned_addr learned my addr 172.21.15.115:0/1224464909 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): get_auth_request con 0x7f91841415c0 auth_method 0 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.564+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f91837fe700 1 --2- 172.21.15.115:0/1224464909 >> v2:172.21.15.157:3300/0 conn(0x7f91840aecc0 0x7f91840a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f91837fe700 10 monclient(hunting): get_auth_request con 0x7f91840aecc0 auth_method 0 2023-09-30T01:04:40.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f91837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f91837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f91837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 -- 172.21.15.115:0/1224464909 >> v2:172.21.15.157:3300/0 conn(0x7f91840aecc0 msgr2=0x7f91840a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 --2- 172.21.15.115:0/1224464909 >> v2:172.21.15.157:3300/0 conn(0x7f91840aecc0 0x7f91840a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 -- 172.21.15.115:0/1224464909 >> v1:172.21.15.157:6789/0 conn(0x7f91840ae8f0 legacy=0x7f91840b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 10 monclient: _finish_hunting 0 2023-09-30T01:04:40.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 monclient: found mon.noname-b 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 -- 172.21.15.115:0/1224464909 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184141a40 con 0x7f91841415c0 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9188e51700 1 --2- 172.21.15.115:0/1224464909 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f917c009f20 tx=0x7f917c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ca4ebe84c649a27 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 1 -- 172.21.15.115:0/1224464909 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00e070 con 0x7f91841415c0 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: dump: 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.577 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: _finish_auth 0 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 1 -- 172.21.15.115:0/1224464909 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c00ba80 con 0x7f91841415c0 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 5 monclient: authenticate success, global_id 4391 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:40.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 1 -- 172.21.15.115:0/1224464909 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0092e0 con 0x7f91841415c0 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 4 set_mon_vals no callback set 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f9182ffd700 10 monclient: dump: 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:40.579 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 10 monclient: shutdown 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 1 -- 172.21.15.115:0/1224464909 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 msgr2=0x7f91840adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.565+0000 7f918a8b4700 1 --2- 172.21.15.115:0/1224464909 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f917c009f20 tx=0x7f917c009840 comp rx=0 tx=0).stop 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 -- 172.21.15.115:0/1224464909 shutdown_connections 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 --2- 172.21.15.115:0/1224464909 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91841415c0 0x7f91840adb60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 --2- 172.21.15.115:0/1224464909 >> v2:172.21.15.157:3300/0 conn(0x7f91840aecc0 0x7f91840a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 -- 172.21.15.115:0/1224464909 >> 172.21.15.115:0/1224464909 conn(0x7f918406a460 msgr2=0x7f91840630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 -- 172.21.15.115:0/1224464909 shutdown_connections 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 -- 172.21.15.115:0/1224464909 wait complete. 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 10 monclient: build_initial_monmap 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 10 monclient: monmap: 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.581 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 librados: starting msgr at 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 librados: starting objecter 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.566+0000 7f918a8b4700 1 Processor -- start 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 -- start start 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 librados: setting wanted keys 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 librados: calling monclient init 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient: init 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:40.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-c con 0x7f91840ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-a con 0x7f91840aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91841415c0 0x7f91841510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): picked mon.noname-b con 0x7f91841415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91840a9e20 con 0x7f91840aecc0 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91840ab540 con 0x7f91841415c0 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:40.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91840a9a40 con 0x7f91840ae8f0 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f918a8b4700 10 monclient(hunting): authenticate will time out at 1142.017760s 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38064/0 (socket says 172.21.15.115:38064) 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:36016/0 (socket says 172.21.15.115:36016) 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 1 -- 172.21.15.115:0/3848762691 learned_addr learned my addr 172.21.15.115:0/3848762691 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): get_auth_request con 0x7f91840ae8f0 auth_method 0 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 10 monclient(hunting): get_auth_request con 0x7f91840aecc0 auth_method 0 2023-09-30T01:04:40.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f91837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9188e51700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91841415c0 0x7f91841510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9188e51700 10 monclient(hunting): get_auth_request con 0x7f91841415c0 auth_method 0 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9188e51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9188e51700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9188e51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.567+0000 7f9183fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f91837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:40.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f91837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f91837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9188e51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 -- 172.21.15.115:0/3848762691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91841415c0 msgr2=0x7f91841510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91841415c0 0x7f91841510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 -- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 msgr2=0x7f9184150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 10 monclient: _finish_hunting 0 2023-09-30T01:04:40.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 monclient: found mon.noname-c 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 -- 172.21.15.115:0/3848762691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c009ae0 con 0x7f91840ae8f0 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f91837fe700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f9183fff700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c590 tx=0x7f916c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 client.?.objecter ms_handle_connect 0x7f91840ae8f0 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c011590 con 0x7f91840ae8f0 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: dump: 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:40.587 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: _finish_auth 0 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 5 monclient: authenticate success, global_id 4394 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c011710 con 0x7f91840ae8f0 2023-09-30T01:04:40.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 10 monclient: _renew_subs 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918413ebe0 con 0x7f91840ae8f0 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c011ad0 con 0x7f91840ae8f0 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f917bfff700 10 monclient: dump: 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f91827fc700 4 set_mon_vals no callback set 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 10 monclient: _renew_subs 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918413e320 con 0x7f91840ae8f0 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.568+0000 7f918a8b4700 1 librados: init done 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.569+0000 7f918a8b4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:40.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.569+0000 7f918a8b4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.569+0000 7f918a8b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.569+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9184072960 con 0x7f91840ae8f0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f916c01b3b0 con 0x7f91840ae8f0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 1 --2- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 0x7f916406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f916c0836d0 con 0x7f91840ae8f0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 10 client.4394.objecter ms_dispatch 0x7f9184001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 3 client.4394.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f917bfff700 3 client.4394.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.570+0000 7f91837fe700 1 --2- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 0x7f916406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.571+0000 7f91837fe700 10 monclient: get_auth_request con 0x7f9164068890 auth_method 0 2023-09-30T01:04:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.571+0000 7f917bfff700 20 client.4394.objecter dump_active .. 0 homeless 2023-09-30T01:04:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.571+0000 7f917bfff700 10 client.4394.objecter ms_handle_connect 0x7f9164068890 2023-09-30T01:04:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.571+0000 7f91837fe700 1 --2- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 0x7f916406ad40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f91840aabd0 tx=0x7f9174008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.574+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f916c051010 con 0x7f91840ae8f0 2023-09-30T01:04:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.574+0000 7f917bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.574+0000 7f917bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:40.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.797+0000 7f918a8b4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:40.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.797+0000 7f918a8b4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:40.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.797+0000 7f918a8b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:40.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.797+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91840aa100 con 0x7f91840ae8f0 2023-09-30T01:04:40.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.798+0000 7f917bfff700 1 -- 172.21.15.115:0/3848762691 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f916c017100 con 0x7f91840ae8f0 2023-09-30T01:04:40.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.798+0000 7f917bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:40.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.798+0000 7f917bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:40.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:40.799 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:40.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 10 librados: watch_flush enter 2023-09-30T01:04:40.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 10 librados: watch_flush exit 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 20 client.4394.objecter shutdown clearing up homeless session... 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 10 client.4394.objecter successfully canceled tick 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 msgr2=0x7f916406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 0x7f916406ad40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f91840aabd0 tx=0x7f9174008040 comp rx=0 tx=0).stop 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 10 monclient: shutdown 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 msgr2=0x7f91840a90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c590 tx=0x7f916c00c8a0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 shutdown_connections 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> v2:172.21.15.115:6816/115635 conn(0x7f9164068890 0x7f916406ad40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91841415c0 0x7f91841510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f91840ae8f0 0x7f91840a90c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 --2- 172.21.15.115:0/3848762691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f91840aecc0 0x7f9184150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 >> 172.21.15.115:0/3848762691 conn(0x7f918406a460 msgr2=0x7f9184144790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 shutdown_connections 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 -- 172.21.15.115:0/3848762691 wait complete. 2023-09-30T01:04:40.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:40.800+0000 7f918a8b4700 1 librados: shutdown 2023-09-30T01:04:43.813 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:43.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.941+0000 7f1787eff700 10 monclient: get_monmap_and_config 2023-09-30T01:04:43.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.941+0000 7f1787eff700 10 monclient: build_initial_monmap 2023-09-30T01:04:43.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.941+0000 7f1787eff700 10 monclient: monmap: 2023-09-30T01:04:43.941 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:43.942324+0000 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:43.942324+0000 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:43.942 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.941+0000 7f1787eff700 1 Processor -- start 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 1 -- start start 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient: init 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17800ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17800aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17800a5d10 0x7f17800adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:43.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-c con 0x7f17800a5d10 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17800a6190 con 0x7f17800ae8f0 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17800a7040 con 0x7f17800aecc0 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1780141cd0 con 0x7f17800a5d10 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1787eff700 10 monclient(hunting): authenticate will time out at 1145.392791s 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1785c9b700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f17800ae8f0 0x7f17800b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44212/0 (socket says 172.21.15.115:44212) 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.942+0000 7f1785c9b700 1 -- 172.21.15.115:0/518470620 learned_addr learned my addr 172.21.15.115:0/518470620 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 --2- 172.21.15.115:0/518470620 >> v2:172.21.15.157:3300/0 conn(0x7f17800a5d10 0x7f17800adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:43.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): get_auth_request con 0x7f17800a5d10 auth_method 0 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055092025 0 0) 0x7f1770001110 con 0x7f17800ae8f0 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): my global_id is 4521 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c003570 con 0x7f17800ae8f0 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115232329 0 0) 0x7f17800a7040 con 0x7f17800aecc0 2023-09-30T01:04:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient(hunting): my global_id is 4486 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1770001110 con 0x7f17800aecc0 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient(hunting): handle_auth_done global_id 4483 payload 274 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 279325858 0 0) 0x7f176c003570 con 0x7f17800ae8f0 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 -- 172.21.15.115:0/518470620 >> v1:172.21.15.157:6789/0 conn(0x7f17800aecc0 legacy=0x7f17801415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 -- 172.21.15.115:0/518470620 >> v1:172.21.15.115:6789/0 conn(0x7f17800ae8f0 legacy=0x7f17800b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 monclient: found mon.noname-c 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:04:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 -- 172.21.15.115:0/518470620 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17800a57f0 con 0x7f17800a5d10 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f178649c700 1 --2- 172.21.15.115:0/518470620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800a5d10 0x7f17800adb60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1778009e60 tx=0x7f1778009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db85ceb26d7688c3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177800e070 con 0x7f17800a5d10 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: dump: 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: _finish_auth 0 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177800b7a0 con 0x7f17800a5d10 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1787eff700 5 monclient: authenticate success, global_id 4483 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1787eff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 1 -- 172.21.15.115:0/518470620 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770001600 con 0x7f17800a5d10 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1787eff700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1787eff700 4 set_mon_vals no callback set 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.943+0000 7f1784c99700 10 monclient: dump: 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 10 monclient: shutdown 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 -- 172.21.15.115:0/518470620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800a5d10 msgr2=0x7f17800adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 --2- 172.21.15.115:0/518470620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800a5d10 0x7f17800adb60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1778009e60 tx=0x7f1778009780 comp rx=0 tx=0).stop 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 -- 172.21.15.115:0/518470620 shutdown_connections 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 --2- 172.21.15.115:0/518470620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800a5d10 0x7f17800adb60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 -- 172.21.15.115:0/518470620 >> 172.21.15.115:0/518470620 conn(0x7f178006a460 msgr2=0x7f17800630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 -- 172.21.15.115:0/518470620 shutdown_connections 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.944+0000 7f1787eff700 1 -- 172.21.15.115:0/518470620 wait complete. 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 10 monclient: build_initial_monmap 2023-09-30T01:04:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 10 monclient: monmap: 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 librados: starting msgr at 2023-09-30T01:04:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 librados: starting objecter 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 Processor -- start 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 -- start start 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 librados: setting wanted keys 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 1 librados: calling monclient init 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 10 monclient: init 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.945+0000 7f1787eff700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-c con 0x7f17800a5d10 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800ae8f0 0x7f17800a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-b con 0x7f17800ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f17800abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): picked mon.noname-a con 0x7f17800aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17800a8640 con 0x7f17800aecc0 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17800ab660 con 0x7f17800ae8f0 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17800ab8d0 con 0x7f17800a5d10 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient(hunting): authenticate will time out at 1145.396072s 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f17800abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44700/0 (socket says 172.21.15.115:44700) 2023-09-30T01:04:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): get_auth_request con 0x7f17800aecc0 auth_method 0 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 1 -- 172.21.15.115:0/1222137665 learned_addr learned my addr 172.21.15.115:0/1222137665 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): get_auth_request con 0x7f17800a5d10 auth_method 0 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient(hunting): handle_auth_done global_id 4524 payload 824 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 -- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 msgr2=0x7f178013e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 -- 172.21.15.115:0/1222137665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800ae8f0 msgr2=0x7f17800a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800ae8f0 0x7f17800a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient: _finish_hunting 0 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 monclient: found mon.noname-a 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:43.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 -- 172.21.15.115:0/1222137665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778009a20 con 0x7f17800aecc0 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1785c9b700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f178649c700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f17800abeb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f17800b59c0 tx=0x7f1778017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 client.?.objecter ms_handle_connect 0x7f17800aecc0 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778012cb0 con 0x7f17800aecc0 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: dump: 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.964 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: _finish_auth 0 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 5 monclient: authenticate success, global_id 4524 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778012e10 con 0x7f17800aecc0 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient: _renew_subs 2023-09-30T01:04:43.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17800a81d0 con 0x7f17800aecc0 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177801e680 con 0x7f17800aecc0 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1775ffb700 10 monclient: dump: 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:43.966 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.946+0000 7f1777fff700 4 set_mon_vals no callback set 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 monclient: _renew_subs 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17800ac5b0 con 0x7f17800aecc0 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 1 librados: init done 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:43.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.947+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17800a83b0 con 0x7f17800aecc0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.948+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f177801e9f0 con 0x7f17800aecc0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 1 --2- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 0x7f176406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1778086490 con 0x7f17800aecc0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 10 client.4524.objecter ms_dispatch 0x7f1780001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 3 client.4524.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 3 client.4524.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1785c9b700 1 --2- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 0x7f176406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1785c9b700 10 monclient: get_auth_request con 0x7f1764068890 auth_method 0 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 20 client.4524.objecter dump_active .. 0 homeless 2023-09-30T01:04:43.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1775ffb700 10 client.4524.objecter ms_handle_connect 0x7f1764068890 2023-09-30T01:04:43.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.949+0000 7f1785c9b700 1 --2- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 0x7f176406ad40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1770007fd0 tx=0x7f1770001330 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:43.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.952+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1778053dd0 con 0x7f17800aecc0 2023-09-30T01:04:43.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.952+0000 7f1775ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:43.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:43.952+0000 7f1775ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:44.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1787eff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:44.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1787eff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:44.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1787eff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:44.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17800aba90 con 0x7f17800aecc0 2023-09-30T01:04:44.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1775ffb700 1 -- 172.21.15.115:0/1222137665 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1778016100 con 0x7f17800aecc0 2023-09-30T01:04:44.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1775ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:44.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.175+0000 7f1775ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:44.176 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:44.176 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 10 librados: watch_flush enter 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 10 librados: watch_flush exit 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 20 client.4524.objecter shutdown clearing up homeless session... 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 10 client.4524.objecter successfully canceled tick 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 msgr2=0x7f176406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 0x7f176406ad40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1770007fd0 tx=0x7f1770001330 comp rx=0 tx=0).stop 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 10 monclient: shutdown 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:44.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 msgr2=0x7f17800abeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f17800abeb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f17800b59c0 tx=0x7f1778017000 comp rx=0 tx=0).stop 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 shutdown_connections 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> v2:172.21.15.115:6816/115635 conn(0x7f1764068890 0x7f176406ad40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f17800abeb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800a5d10 0x7f178013e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 --2- 172.21.15.115:0/1222137665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17800ae8f0 0x7f17800a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 >> 172.21.15.115:0/1222137665 conn(0x7f178006a460 msgr2=0x7f178013f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 shutdown_connections 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 -- 172.21.15.115:0/1222137665 wait complete. 2023-09-30T01:04:44.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:44.177+0000 7f1787eff700 1 librados: shutdown 2023-09-30T01:04:47.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:47.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.318+0000 7fa31b451700 10 monclient: get_monmap_and_config 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.318+0000 7fa31b451700 10 monclient: build_initial_monmap 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.318+0000 7fa31b451700 10 monclient: monmap: 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:47.319763+0000 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:47.319763+0000 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:47.319 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:47.320 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:47.320 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:47.320 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:47.320 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.319+0000 7fa31b451700 1 Processor -- start 2023-09-30T01:04:47.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.319+0000 7fa31b451700 1 -- start start 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.319+0000 7fa31b451700 10 monclient: init 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3140ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3140aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa3140a5d10 0x7fa3140adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-a con 0x7fa3140a5d10 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140a6190 con 0x7fa3140aecc0 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140a7040 con 0x7fa3140ae8f0 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa314141cd0 con 0x7fa3140a5d10 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa31b451700 10 monclient(hunting): authenticate will time out at 1148.770469s 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3189ec700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa3140aecc0 0x7fa3141415c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44218/0 (socket says 172.21.15.115:44218) 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3189ec700 1 -- 172.21.15.115:0/652995435 learned_addr learned my addr 172.21.15.115:0/652995435 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 1 --2- 172.21.15.115:0/652995435 >> v2:172.21.15.115:3300/0 conn(0x7fa3140a5d10 0x7fa3140adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): get_auth_request con 0x7fa3140a5d10 auth_method 0 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:47.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient(hunting): handle_auth_done global_id 4527 payload 274 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 1 -- 172.21.15.115:0/652995435 >> v1:172.21.15.157:6789/0 conn(0x7fa3140ae8f0 legacy=0x7fa3140b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 1 -- 172.21.15.115:0/652995435 >> v1:172.21.15.115:6789/0 conn(0x7fa3140aecc0 legacy=0x7fa3141415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient: _finish_hunting 0 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 1 monclient: found mon.noname-a 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa30bfff700 1 -- 172.21.15.115:0/652995435 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176426033 0 0) 0x7fa310001110 con 0x7fa3140aecc0 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa3199ee700 1 -- 172.21.15.115:0/652995435 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140a57f0 con 0x7fa3140a5d10 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.320+0000 7fa30bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa3199ee700 1 --2- 172.21.15.115:0/652995435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140a5d10 0x7fa3140adb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009e60 tx=0x7fa30c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5b4b1753d480e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 1 -- 172.21.15.115:0/652995435 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00e070 con 0x7fa3140a5d10 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: dump: 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.332 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: _finish_auth 0 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 5 monclient: authenticate success, global_id 4527 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 1 -- 172.21.15.115:0/652995435 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00bc30 con 0x7fa3140a5d10 2023-09-30T01:04:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 4 set_mon_vals no callback set 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 1 -- 172.21.15.115:0/652995435 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c012430 con 0x7fa3140a5d10 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa30bfff700 10 monclient: dump: 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:47.334 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 10 monclient: shutdown 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 -- 172.21.15.115:0/652995435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140a5d10 msgr2=0x7fa3140adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 --2- 172.21.15.115:0/652995435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140a5d10 0x7fa3140adb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009e60 tx=0x7fa30c009780 comp rx=0 tx=0).stop 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 -- 172.21.15.115:0/652995435 shutdown_connections 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 --2- 172.21.15.115:0/652995435 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140a5d10 0x7fa3140adb60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 -- 172.21.15.115:0/652995435 >> 172.21.15.115:0/652995435 conn(0x7fa31406a460 msgr2=0x7fa3140630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 -- 172.21.15.115:0/652995435 shutdown_connections 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.321+0000 7fa31b451700 1 -- 172.21.15.115:0/652995435 wait complete. 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 10 monclient: build_initial_monmap 2023-09-30T01:04:47.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 10 monclient: monmap: 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 librados: starting msgr at 2023-09-30T01:04:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 librados: starting objecter 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 Processor -- start 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 -- start start 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 librados: setting wanted keys 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 1 librados: calling monclient init 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.322+0000 7fa31b451700 10 monclient: init 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 0x7fa31413e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-c con 0x7fa3140a5d10 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3140ae8f0 0x7fa3140a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-b con 0x7fa3140ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:47.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): picked mon.noname-a con 0x7fa3140aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3140a9e40 con 0x7fa3140aecc0 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3140a7d10 con 0x7fa3140ae8f0 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3140ab940 con 0x7fa3140a5d10 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa31b451700 10 monclient(hunting): authenticate will time out at 1148.773581s 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:47.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57494/0 (socket says 172.21.15.115:57494) 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 -- 172.21.15.115:0/2646111517 learned_addr learned my addr 172.21.15.115:0/2646111517 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 0x7fa31413e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): get_auth_request con 0x7fa3140aecc0 auth_method 0 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): get_auth_request con 0x7fa3140a5d10 auth_method 0 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:47.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3140ae8f0 0x7fa3140a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): get_auth_request con 0x7fa3140ae8f0 auth_method 0 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3191ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3189ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient(hunting): handle_auth_done global_id 4533 payload 824 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 -- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 msgr2=0x7fa31413e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 0x7fa31413e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 -- 172.21.15.115:0/2646111517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3140ae8f0 msgr2=0x7fa3140a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3140ae8f0 0x7fa3140a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 10 monclient: _finish_hunting 0 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 1 monclient: found mon.noname-a 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.323+0000 7fa3199ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3199ee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3199ee700 1 -- 172.21.15.115:0/2646111517 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c009a20 con 0x7fa3140aecc0 2023-09-30T01:04:47.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3191ed700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 0x7fa31413e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3199ee700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa30c013000 tx=0x7fa30c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 client.?.objecter ms_handle_connect 0x7fa3140aecc0 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c01b420 con 0x7fa3140aecc0 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: dump: 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.343 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: _finish_auth 0 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c01b580 con 0x7fa3140aecc0 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 5 monclient: authenticate success, global_id 4533 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 10 monclient: _renew_subs 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3140aa810 con 0x7fa3140aecc0 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c01b920 con 0x7fa3140aecc0 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa3097fa700 10 monclient: dump: 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:47.345 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa30b7fe700 4 set_mon_vals no callback set 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 10 monclient: _renew_subs 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3140a9590 con 0x7fa3140aecc0 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.324+0000 7fa31b451700 1 librados: init done 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.325+0000 7fa31b451700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.325+0000 7fa31b451700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:47.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.325+0000 7fa31b451700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.325+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3140a9ff0 con 0x7fa3140aecc0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.325+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa30c01bc90 con 0x7fa3140aecc0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 1 --2- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 0x7fa2f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3191ed700 1 --2- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 0x7fa2f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa30c085cc0 con 0x7fa3140aecc0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 10 client.4533.objecter ms_dispatch 0x7fa314001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 3 client.4533.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 3 client.4533.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3191ed700 10 monclient: get_auth_request con 0x7fa2f8068890 auth_method 0 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 20 client.4533.objecter dump_active .. 0 homeless 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3097fa700 10 client.4533.objecter ms_handle_connect 0x7fa2f8068890 2023-09-30T01:04:47.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.326+0000 7fa3191ed700 1 --2- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 0x7fa2f806ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa304001dc0 tx=0x7fa304009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:47.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.330+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3140a9ff0 con 0x7fa3140aecc0 2023-09-30T01:04:47.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.330+0000 7fa3097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:47.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.330+0000 7fa3097fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:47.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.554+0000 7fa31b451700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.554+0000 7fa31b451700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.554+0000 7fa31b451700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.554+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa3140a9ff0 con 0x7fa3140aecc0 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.555+0000 7fa3097fa700 1 -- 172.21.15.115:0/2646111517 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa30c02a100 con 0x7fa3140aecc0 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.555+0000 7fa3097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.555+0000 7fa3097fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:47.557 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:47.558 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 10 librados: watch_flush enter 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 10 librados: watch_flush exit 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 20 client.4533.objecter shutdown clearing up homeless session... 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 10 client.4533.objecter successfully canceled tick 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 msgr2=0x7fa2f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 0x7fa2f806ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa304001dc0 tx=0x7fa304009040 comp rx=0 tx=0).stop 2023-09-30T01:04:47.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 10 monclient: shutdown 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 msgr2=0x7fa3140aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa30c013000 tx=0x7fa30c019000 comp rx=0 tx=0).stop 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 shutdown_connections 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> v2:172.21.15.115:6816/115635 conn(0x7fa2f8068890 0x7fa2f806ad40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa3140aecc0 0x7fa3140aa2d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa3140a5d10 0x7fa31413e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 --2- 172.21.15.115:0/2646111517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3140ae8f0 0x7fa3140a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 >> 172.21.15.115:0/2646111517 conn(0x7fa31406a460 msgr2=0x7fa3140b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 shutdown_connections 2023-09-30T01:04:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 -- 172.21.15.115:0/2646111517 wait complete. 2023-09-30T01:04:47.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:47.557+0000 7fa31b451700 1 librados: shutdown 2023-09-30T01:04:50.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:50.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 10 monclient: get_monmap_and_config 2023-09-30T01:04:50.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 10 monclient: build_initial_monmap 2023-09-30T01:04:50.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 10 monclient: monmap: 2023-09-30T01:04:50.695 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:50.695 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:50.696142+0000 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:50.696142+0000 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:50.696 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 1 Processor -- start 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 1 -- start start 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.695+0000 7f1431be9700 10 monclient: init 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f142c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f142c0aecc0 0x7f142c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-a con 0x7f142c0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f142c1415c0 0x7f142c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-c con 0x7f142c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c0a7040 con 0x7f142c0ae8f0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c0a57f0 con 0x7f142c0aecc0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c0a5480 con 0x7f142c1415c0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f1431be9700 10 monclient(hunting): authenticate will time out at 1152.146596s 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f142c0aecc0 0x7f142c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142b7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f142c0ae8f0 0x7f142c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44222/0 (socket says 172.21.15.115:44222) 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142b7fe700 1 -- 172.21.15.115:0/1382228403 learned_addr learned my addr 172.21.15.115:0/1382228403 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): get_auth_request con 0x7f142c0aecc0 auth_method 0 2023-09-30T01:04:50.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142bfff700 1 --2- 172.21.15.115:0/1382228403 >> v2:172.21.15.157:3300/0 conn(0x7f142c1415c0 0x7f142c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142bfff700 10 monclient(hunting): get_auth_request con 0x7f142c1415c0 auth_method 0 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:50.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 10 monclient(hunting): handle_auth_done global_id 4536 payload 274 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 1 -- 172.21.15.115:0/1382228403 >> v2:172.21.15.157:3300/0 conn(0x7f142c1415c0 msgr2=0x7f142c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.696+0000 7f142affd700 1 --2- 172.21.15.115:0/1382228403 >> v2:172.21.15.157:3300/0 conn(0x7f142c1415c0 0x7f142c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 1 -- 172.21.15.115:0/1382228403 >> v1:172.21.15.115:6789/0 conn(0x7f142c0ae8f0 legacy=0x7f142c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 10 monclient: _finish_hunting 0 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 1 monclient: found mon.noname-a 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 1 -- 172.21.15.115:0/1382228403 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174725721 0 0) 0x7f142c0a7040 con 0x7f142c0ae8f0 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 1 -- 172.21.15.115:0/1382228403 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c0a55c0 con 0x7f142c0aecc0 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:04:50.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142affd700 1 --2- 172.21.15.115:0/1382228403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c0aecc0 0x7f142c0a5d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f142c0a6250 tx=0x7f14140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b80dad8e33db22f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 1 -- 172.21.15.115:0/1382228403 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400b6b0 con 0x7f142c0aecc0 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: dump: 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:50.710 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 1 -- 172.21.15.115:0/1382228403 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400b810 con 0x7f142c0aecc0 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 5 monclient: authenticate success, global_id 4536 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 1 -- 172.21.15.115:0/1382228403 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400bbb0 con 0x7f142c0aecc0 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:50.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 4 set_mon_vals no callback set 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f142a7fc700 10 monclient: dump: 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:50.712 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 10 monclient: shutdown 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 -- 172.21.15.115:0/1382228403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c0aecc0 msgr2=0x7f142c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 --2- 172.21.15.115:0/1382228403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c0aecc0 0x7f142c0a5d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f142c0a6250 tx=0x7f14140091a0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 -- 172.21.15.115:0/1382228403 shutdown_connections 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 --2- 172.21.15.115:0/1382228403 >> v2:172.21.15.157:3300/0 conn(0x7f142c1415c0 0x7f142c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 --2- 172.21.15.115:0/1382228403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c0aecc0 0x7f142c0a5d10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 -- 172.21.15.115:0/1382228403 >> 172.21.15.115:0/1382228403 conn(0x7f142c06a460 msgr2=0x7f142c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 -- 172.21.15.115:0/1382228403 shutdown_connections 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.697+0000 7f1431be9700 1 -- 172.21.15.115:0/1382228403 wait complete. 2023-09-30T01:04:50.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.698+0000 7f1431be9700 10 monclient: build_initial_monmap 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.698+0000 7f1431be9700 10 monclient: monmap: 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:50.714 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 librados: starting msgr at 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 librados: starting objecter 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 Processor -- start 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 -- start start 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 librados: setting wanted keys 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 1 librados: calling monclient init 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.699+0000 7f1431be9700 10 monclient: init 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f142c0ae8f0 0x7f142c1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-b con 0x7f142c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 0x7f142c13e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-c con 0x7f142c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): picked mon.noname-a con 0x7f142c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0ab820 con 0x7f142c1415c0 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0a9d10 con 0x7f142c0ae8f0 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0aba20 con 0x7f142c0aecc0 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:50.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient(hunting): authenticate will time out at 1152.150199s 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 0x7f142c13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57514/0 (socket says 172.21.15.115:57514) 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 -- 172.21.15.115:0/4285289314 learned_addr learned my addr 172.21.15.115:0/4285289314 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): get_auth_request con 0x7f142c1415c0 auth_method 0 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): get_auth_request con 0x7f142c0aecc0 auth_method 0 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:50.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f142c0ae8f0 0x7f142c1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): get_auth_request con 0x7f142c0ae8f0 auth_method 0 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:50.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient(hunting): handle_auth_done global_id 4542 payload 824 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 -- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 msgr2=0x7f142c13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 0x7f142c13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 -- 172.21.15.115:0/4285289314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f142c0ae8f0 msgr2=0x7f142c1511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f142c0ae8f0 0x7f142c1511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 monclient: found mon.noname-a 2023-09-30T01:04:50.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 -- 172.21.15.115:0/4285289314 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009420 con 0x7f142c1415c0 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142affd700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 0x7f142c13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f142bfff700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1424007f20 tx=0x7f1424009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 client.?.objecter ms_handle_connect 0x7f142c1415c0 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400eb90 con 0x7f142c1415c0 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: dump: 2023-09-30T01:04:50.720 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: _finish_auth 0 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 5 monclient: authenticate success, global_id 4542 2023-09-30T01:04:50.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142400ed10 con 0x7f142c1415c0 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424014670 con 0x7f142c1415c0 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _renew_subs 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c0a9900 con 0x7f142c1415c0 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1429ffb700 4 set_mon_vals no callback set 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f14237fe700 10 monclient: dump: 2023-09-30T01:04:50.722 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:50.723 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _renew_subs 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c0aa3d0 con 0x7f142c1415c0 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.700+0000 7f1431be9700 1 librados: init done 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f14240149e0 con 0x7f142c1415c0 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f1431be9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f1431be9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f1431be9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c0a9f30 con 0x7f142c1415c0 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 1 --2- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 0x7f140c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:50.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1424084450 con 0x7f142c1415c0 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 10 client.4542.objecter ms_dispatch 0x7f142c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 3 client.4542.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 3 client.4542.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f142b7fe700 1 --2- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 0x7f140c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f142b7fe700 10 monclient: get_auth_request con 0x7f140c068890 auth_method 0 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.702+0000 7f14237fe700 20 client.4542.objecter dump_active .. 0 homeless 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.703+0000 7f14237fe700 10 client.4542.objecter ms_handle_connect 0x7f140c068890 2023-09-30T01:04:50.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.703+0000 7f142b7fe700 1 --2- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 0x7f140c06ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f141c007fd0 tx=0x7f141c007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:50.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.707+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f142c0a9f30 con 0x7f142c1415c0 2023-09-30T01:04:50.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.707+0000 7f14237fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:50.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.707+0000 7f14237fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:50.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f1431be9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:50.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f1431be9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f1431be9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f142c0a9f30 con 0x7f142c1415c0 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f14237fe700 1 -- 172.21.15.115:0/4285289314 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f142c0a9f30 con 0x7f142c1415c0 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f14237fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.927+0000 7f14237fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:50.928 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 10 librados: watch_flush enter 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 10 librados: watch_flush exit 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 20 client.4542.objecter shutdown clearing up homeless session... 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 10 client.4542.objecter successfully canceled tick 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 msgr2=0x7f140c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 0x7f140c06ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f141c007fd0 tx=0x7f141c007d30 comp rx=0 tx=0).stop 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 10 monclient: shutdown 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 msgr2=0x7f142c0a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1424007f20 tx=0x7f1424009860 comp rx=0 tx=0).stop 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 shutdown_connections 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> v2:172.21.15.115:6816/115635 conn(0x7f140c068890 0x7f140c06ad40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f142c1415c0 0x7f142c0a93c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f142c0ae8f0 0x7f142c1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 --2- 172.21.15.115:0/4285289314 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f142c0aecc0 0x7f142c13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 >> 172.21.15.115:0/4285289314 conn(0x7f142c06a460 msgr2=0x7f142c0b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:50.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 shutdown_connections 2023-09-30T01:04:50.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 -- 172.21.15.115:0/4285289314 wait complete. 2023-09-30T01:04:50.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:50.929+0000 7f1431be9700 1 librados: shutdown 2023-09-30T01:04:53.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 10 monclient: get_monmap_and_config 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 10 monclient: build_initial_monmap 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 10 monclient: monmap: 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:54.073081+0000 2023-09-30T01:04:54.072 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:54.073081+0000 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:54.073 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 1 Processor -- start 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 1 -- start start 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.071+0000 7f74a6301700 10 monclient: init 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-b con 0x7f74a00ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:54.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 0x7f74a00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-c con 0x7f74a00aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74a01415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a00a7040 con 0x7f74a01415c0 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a00b19b0 con 0x7f74a00aecc0 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00a57f0 con 0x7f74a00ae8f0 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f74a6301700 10 monclient(hunting): authenticate will time out at 1155.523555s 2023-09-30T01:04:54.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36468/0 (socket says 172.21.15.115:36468) 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 -- 172.21.15.115:0/1409027652 learned_addr learned my addr 172.21.15.115:0/1409027652 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): get_auth_request con 0x7f74a00ae8f0 auth_method 0 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749f7fe700 1 --2- 172.21.15.115:0/1409027652 >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 0x7f74a00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 -- 172.21.15.115:0/1409027652 >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 msgr2=0x7f74a00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:54.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 --2- 172.21.15.115:0/1409027652 >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 0x7f74a00b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 1 -- 172.21.15.115:0/1409027652 >> v1:172.21.15.157:6789/0 conn(0x7f74a01415c0 legacy=0x7f74a00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.072+0000 7f749ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749ffff700 1 monclient: found mon.noname-b 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749ffff700 1 -- 172.21.15.115:0/1409027652 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a00a5480 con 0x7f74a00ae8f0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749f7fe700 10 monclient: get_auth_request con 0x7f74a00aecc0 auth_method 0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749f7fe700 1 --2- 172.21.15.115:0/1409027652 >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 0x7f74a00b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749ffff700 1 --2- 172.21.15.115:0/1409027652 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74900099b0 tx=0x7f7490009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cbaae31551b1486 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 1 -- 172.21.15.115:0/1409027652 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749000b820 con 0x7f74a00ae8f0 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:54.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: dump: 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: _finish_auth 0 2023-09-30T01:04:54.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 1 -- 172.21.15.115:0/1409027652 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749000b980 con 0x7f74a00ae8f0 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 5 monclient: authenticate success, global_id 4406 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 1 -- 172.21.15.115:0/1409027652 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749000bd20 con 0x7f74a00ae8f0 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 4 set_mon_vals no callback set 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f749effd700 10 monclient: dump: 2023-09-30T01:04:54.086 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 10 monclient: shutdown 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 -- 172.21.15.115:0/1409027652 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 msgr2=0x7f74a00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:54.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 --2- 172.21.15.115:0/1409027652 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74900099b0 tx=0x7f7490009250 comp rx=0 tx=0).stop 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 -- 172.21.15.115:0/1409027652 shutdown_connections 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 --2- 172.21.15.115:0/1409027652 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00ae8f0 0x7f74a00a5d10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 --2- 172.21.15.115:0/1409027652 >> v2:172.21.15.157:3300/0 conn(0x7f74a00aecc0 0x7f74a00b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 -- 172.21.15.115:0/1409027652 >> 172.21.15.115:0/1409027652 conn(0x7f74a006a460 msgr2=0x7f74a00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 -- 172.21.15.115:0/1409027652 shutdown_connections 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.073+0000 7f74a6301700 1 -- 172.21.15.115:0/1409027652 wait complete. 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.074+0000 7f74a6301700 10 monclient: build_initial_monmap 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.074+0000 7f74a6301700 10 monclient: monmap: 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.074+0000 7f74a6301700 1 librados: starting msgr at 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.074+0000 7f74a6301700 1 librados: starting objecter 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.074+0000 7f74a6301700 1 Processor -- start 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 -- start start 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 librados: setting wanted keys 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 librados: calling monclient init 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient: init 2023-09-30T01:04:54.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-a con 0x7f74a00ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-c con 0x7f74a00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74a01415c0 0x7f74a014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): picked mon.noname-b con 0x7f74a01415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00a9a90 con 0x7f74a00ae8f0 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:54.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00a9770 con 0x7f74a01415c0 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00aaac0 con 0x7f74a00aecc0 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a6301700 10 monclient(hunting): authenticate will time out at 1155.526616s 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50444/0 (socket says 172.21.15.115:50444) 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36484/0 (socket says 172.21.15.115:36484) 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 1 -- 172.21.15.115:0/3563776203 learned_addr learned my addr 172.21.15.115:0/3563776203 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): get_auth_request con 0x7f74a00ae8f0 auth_method 0 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:54.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): get_auth_request con 0x7f74a00aecc0 auth_method 0 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74a01415c0 0x7f74a014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): get_auth_request con 0x7f74a01415c0 auth_method 0 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f74a489e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.075+0000 7f749f7fe700 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 -- 172.21.15.115:0/3563776203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74a01415c0 msgr2=0x7f74a014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74a01415c0 0x7f74a014ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 -- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 msgr2=0x7f74a00a9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:54.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 monclient: found mon.noname-c 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 -- 172.21.15.115:0/3563776203 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900094f0 con 0x7f74a00aecc0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749ffff700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749f7fe700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f749400c880 tx=0x7f7494008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 client.?.objecter ms_handle_connect 0x7f74a00aecc0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494011600 con 0x7f74a00aecc0 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:54.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: dump: 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: _finish_auth 0 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 5 monclient: authenticate success, global_id 4409 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7494011760 con 0x7f74a00aecc0 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494011ae0 con 0x7f74a00aecc0 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 10 monclient: _renew_subs 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a00a8f80 con 0x7f74a00aecc0 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f749e7fc700 4 set_mon_vals no callback set 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f7487fff700 10 monclient: dump: 2023-09-30T01:04:54.096 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 10 monclient: _renew_subs 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a014e5e0 con 0x7f74a00aecc0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.076+0000 7f74a6301700 1 librados: init done 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.077+0000 7f74a6301700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.077+0000 7f74a6301700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.077+0000 7f74a6301700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.077+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a00a9bd0 con 0x7f74a00aecc0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.077+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7494022350 con 0x7f74a00aecc0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 1 --2- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 0x7f748c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f749ffff700 1 --2- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 0x7f748c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7494083d70 con 0x7f74a00aecc0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 10 client.4409.objecter ms_dispatch 0x7f74a0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 3 client.4409.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 3 client.4409.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f749ffff700 10 monclient: get_auth_request con 0x7f748c068890 auth_method 0 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 20 client.4409.objecter dump_active .. 0 homeless 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f7487fff700 10 client.4409.objecter ms_handle_connect 0x7f748c068890 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.078+0000 7f749ffff700 1 --2- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 0x7f748c06ad40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f749000c010 tx=0x7f7490016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.083+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74a00a9bd0 con 0x7f74a00aecc0 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.083+0000 7f7487fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.083+0000 7f7487fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:54.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f74a6301700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f74a6301700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f74a6301700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74a00a9bd0 con 0x7f74a00aecc0 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f7487fff700 1 -- 172.21.15.115:0/3563776203 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f749401f100 con 0x7f74a00aecc0 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f7487fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.303+0000 7f7487fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:54.304 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 10 librados: watch_flush enter 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 10 librados: watch_flush exit 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 20 client.4409.objecter shutdown clearing up homeless session... 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 10 client.4409.objecter successfully canceled tick 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 msgr2=0x7f748c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 0x7f748c06ad40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f749000c010 tx=0x7f7490016000 comp rx=0 tx=0).stop 2023-09-30T01:04:54.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 10 monclient: shutdown 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 msgr2=0x7f74a00a8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f749400c880 tx=0x7f7494008040 comp rx=0 tx=0).stop 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 shutdown_connections 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> v2:172.21.15.115:6816/115635 conn(0x7f748c068890 0x7f748c06ad40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74a01415c0 0x7f74a014ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74a00ae8f0 0x7f74a00a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 --2- 172.21.15.115:0/3563776203 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f74a00aecc0 0x7f74a00a8980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.305+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 >> 172.21.15.115:0/3563776203 conn(0x7f74a006a460 msgr2=0x7f74a013fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.306+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 shutdown_connections 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.306+0000 7f74a6301700 1 -- 172.21.15.115:0/3563776203 wait complete. 2023-09-30T01:04:54.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:54.306+0000 7f74a6301700 1 librados: shutdown 2023-09-30T01:04:57.318 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:04:57.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.446+0000 7fd8bba3f700 10 monclient: get_monmap_and_config 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.446+0000 7fd8bba3f700 10 monclient: build_initial_monmap 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.446+0000 7fd8bba3f700 10 monclient: monmap: 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:04:57.447756+0000 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:04:57.447756+0000 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:57.447 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:04:57.448 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.447+0000 7fd8bba3f700 1 Processor -- start 2023-09-30T01:04:57.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.447+0000 7fd8bba3f700 1 -- start start 2023-09-30T01:04:57.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.447+0000 7fd8bba3f700 10 monclient: init 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 0x7fd8b40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8b40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8b40aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8b41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40a7040 con 0x7fd8b40aecc0 2023-09-30T01:04:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40b19b0 con 0x7fd8b40ae8f0 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40a57f0 con 0x7fd8b41415c0 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8bba3f700 10 monclient(hunting): authenticate will time out at 1158.898244s 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50448/0 (socket says 172.21.15.115:50448) 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36494/0 (socket says 172.21.15.115:36494) 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 1 -- 172.21.15.115:0/766245311 learned_addr learned my addr 172.21.15.115:0/766245311 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): get_auth_request con 0x7fd8b40aecc0 auth_method 0 2023-09-30T01:04:57.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): get_auth_request con 0x7fd8b41415c0 auth_method 0 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:57.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 0x7fd8b40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 10 monclient(hunting): get_auth_request con 0x7fd8b40ae8f0 auth_method 0 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 -- 172.21.15.115:0/766245311 >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 msgr2=0x7fd8b40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 0x7fd8b40a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 -- 172.21.15.115:0/766245311 >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 msgr2=0x7fd8b40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient: _finish_hunting 0 2023-09-30T01:04:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 monclient: found mon.noname-b 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 -- 172.21.15.115:0/766245311 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40a5480 con 0x7fd8b41415c0 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b8fda700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b97db700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 0x7fd8b40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.448+0000 7fd8b9fdc700 1 --2- 172.21.15.115:0/766245311 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40b59c0 tx=0x7fd8ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63797fd26d8916fb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 1 -- 172.21.15.115:0/766245311 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00e070 con 0x7fd8b41415c0 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: dump: 2023-09-30T01:04:57.462 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: _finish_auth 0 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:57.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 1 -- 172.21.15.115:0/766245311 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac00bb60 con 0x7fd8b41415c0 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 5 monclient: authenticate success, global_id 4412 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 1 -- 172.21.15.115:0/766245311 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac012430 con 0x7fd8b41415c0 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 10 monclient: get_monmap_and_config success 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 4 set_mon_vals no callback set 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8abfff700 10 monclient: dump: 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.464 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 10 monclient: shutdown 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 -- 172.21.15.115:0/766245311 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 msgr2=0x7fd8b40adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/766245311 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40b59c0 tx=0x7fd8ac009780 comp rx=0 tx=0).stop 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 -- 172.21.15.115:0/766245311 shutdown_connections 2023-09-30T01:04:57.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/766245311 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b41415c0 0x7fd8b40adb60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.157:3300/0 conn(0x7fd8b40ae8f0 0x7fd8b40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/766245311 >> v2:172.21.15.115:3300/0 conn(0x7fd8b40aecc0 0x7fd8b40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 -- 172.21.15.115:0/766245311 >> 172.21.15.115:0/766245311 conn(0x7fd8b406a460 msgr2=0x7fd8b40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 -- 172.21.15.115:0/766245311 shutdown_connections 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.449+0000 7fd8bba3f700 1 -- 172.21.15.115:0/766245311 wait complete. 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 10 monclient: build_initial_monmap 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 10 monclient: monmap: 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:04:57.466 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 librados: starting msgr at 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 librados: starting objecter 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 Processor -- start 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 -- start start 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 librados: setting wanted keys 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 1 librados: calling monclient init 2023-09-30T01:04:57.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.450+0000 7fd8bba3f700 10 monclient: init 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient: _reopen_session rank -1 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8b40ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 0x7fd8b414e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8b40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8b41415c0 0x7fd8b4151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8b41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b414e940 con 0x7fd8b40ae8f0 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40a85d0 con 0x7fd8b41415c0 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40a8790 con 0x7fd8b40aecc0 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): _renew_subs 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8bba3f700 10 monclient(hunting): authenticate will time out at 1158.901643s 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50464/0 (socket says 172.21.15.115:50464) 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 1 -- 172.21.15.115:0/1261384208 learned_addr learned my addr 172.21.15.115:0/1261384208 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): get_auth_request con 0x7fd8b40ae8f0 auth_method 0 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 0x7fd8b414e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): get_auth_request con 0x7fd8b40aecc0 auth_method 0 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b97db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:04:57.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b9fdc700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8b41415c0 0x7fd8b4151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b9fdc700 10 monclient(hunting): get_auth_request con 0x7fd8b41415c0 auth_method 0 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b9fdc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b9fdc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b9fdc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.451+0000 7fd8b8fda700 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 -- 172.21.15.115:0/1261384208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8b41415c0 msgr2=0x7fd8b4151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8b41415c0 0x7fd8b4151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 -- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 msgr2=0x7fd8b413e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 10 monclient: _finish_hunting 0 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 monclient: found mon.noname-c 2023-09-30T01:04:57.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 -- 172.21.15.115:0/1261384208 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00094f0 con 0x7fd8b40aecc0 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b97db700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8b8fda700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 0x7fd8b414e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0006fd0 tx=0x7fd8b0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 client.?.objecter ms_handle_connect 0x7fd8b40aecc0 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000c850 con 0x7fd8b40aecc0 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:57.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: dump: 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.473 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: _finish_auth 0 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:04:57.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b000c9b0 con 0x7fd8b40aecc0 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 5 monclient: authenticate success, global_id 4415 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 10 monclient: _renew_subs 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac009a20 con 0x7fd8b40aecc0 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000cd30 con 0x7fd8b40aecc0 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8a97fa700 10 monclient: dump: 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:04:57.475 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8ab7fe700 4 set_mon_vals no callback set 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 10 monclient: _renew_subs 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b41518d0 con 0x7fd8b40aecc0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.452+0000 7fd8bba3f700 1 librados: init done 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.453+0000 7fd8bba3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.453+0000 7fd8bba3f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.453+0000 7fd8bba3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.453+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b4072960 con 0x7fd8b40aecc0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.453+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd8b0013680 con 0x7fd8b40aecc0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 1 --2- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 0x7fd89806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd8b00840c0 con 0x7fd8b40aecc0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 10 client.4415.objecter ms_dispatch 0x7fd8b4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 3 client.4415.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 3 client.4415.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:04:57.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8b97db700 1 --2- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 0x7fd89806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8b97db700 10 monclient: get_auth_request con 0x7fd898068890 auth_method 0 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 20 client.4415.objecter dump_active .. 0 homeless 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8a97fa700 10 client.4415.objecter ms_handle_connect 0x7fd898068890 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.454+0000 7fd8b97db700 1 --2- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 0x7fd89806ad40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006dc0 tx=0x7fd8a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.458+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8b0051980 con 0x7fd8b40aecc0 2023-09-30T01:04:57.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.458+0000 7fd8a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:04:57.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.458+0000 7fd8a97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:04:57.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.685+0000 7fd8bba3f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:04:57.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.685+0000 7fd8bba3f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:57.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.685+0000 7fd8bba3f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:04:57.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.685+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd8b4141120 con 0x7fd8b40aecc0 2023-09-30T01:04:57.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.686+0000 7fd8a97fa700 1 -- 172.21.15.115:0/1261384208 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd8b0054fa0 con 0x7fd8b40aecc0 2023-09-30T01:04:57.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.686+0000 7fd8a97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:04:57.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.686+0000 7fd8a97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:04:57.688 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:04:57.688 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:04:57.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.687+0000 7fd8bba3f700 10 librados: watch_flush enter 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.687+0000 7fd8bba3f700 10 librados: watch_flush exit 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.687+0000 7fd8bba3f700 20 client.4415.objecter shutdown clearing up homeless session... 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 10 client.4415.objecter successfully canceled tick 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 msgr2=0x7fd89806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 0x7fd89806ad40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4006dc0 tx=0x7fd8a4008040 comp rx=0 tx=0).stop 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 10 monclient: shutdown 2023-09-30T01:04:57.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 msgr2=0x7fd8b414e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 0x7fd8b414e030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0006fd0 tx=0x7fd8b0009250 comp rx=0 tx=0).stop 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 shutdown_connections 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> v2:172.21.15.115:6816/115635 conn(0x7fd898068890 0x7fd89806ad40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8b41415c0 0x7fd8b4151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8b40ae8f0 0x7fd8b413e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 --2- 172.21.15.115:0/1261384208 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8b40aecc0 0x7fd8b414e030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:04:57.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 >> 172.21.15.115:0/1261384208 conn(0x7fd8b406a460 msgr2=0x7fd8b41445b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:04:57.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 shutdown_connections 2023-09-30T01:04:57.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 -- 172.21.15.115:0/1261384208 wait complete. 2023-09-30T01:04:57.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:04:57.688+0000 7fd8bba3f700 1 librados: shutdown 2023-09-30T01:05:00.700 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.828+0000 7f50dc224700 10 monclient: get_monmap_and_config 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.828+0000 7f50dc224700 10 monclient: build_initial_monmap 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.828+0000 7f50dc224700 10 monclient: monmap: 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:00.829239+0000 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:00.829239+0000 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:00.829 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.828+0000 7f50dc224700 1 Processor -- start 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 -- start start 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient: init 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50d40ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-b con 0x7f50d40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-a con 0x7f50d41415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d40a7040 con 0x7f50d40ae8f0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d40a57f0 con 0x7f50d41415c0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40a5480 con 0x7f50d40aecc0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50dc224700 10 monclient(hunting): authenticate will time out at 1162.279657s 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d9fc0700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f50d40ae8f0 0x7f50d40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:48516/0 (socket says 172.21.15.115:48516) 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50478/0 (socket says 172.21.15.115:50478) 2023-09-30T01:05:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36516/0 (socket says 172.21.15.115:36516) 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d9fc0700 1 -- 172.21.15.115:0/143445666 learned_addr learned my addr 172.21.15.115:0/143445666 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): get_auth_request con 0x7f50d40aecc0 auth_method 0 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): get_auth_request con 0x7f50d41415c0 auth_method 0 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:00.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.829+0000 7f50da7c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 -- 172.21.15.115:0/143445666 >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 msgr2=0x7f50d40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 --2- 172.21.15.115:0/143445666 >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 -- 172.21.15.115:0/143445666 >> v1:172.21.15.115:6789/0 conn(0x7f50d40ae8f0 legacy=0x7f50d40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 10 monclient: _finish_hunting 0 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 monclient: found mon.noname-b 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:00.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 -- 172.21.15.115:0/143445666 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40a55c0 con 0x7f50d40aecc0 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 1 -- 172.21.15.115:0/143445666 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278923091 0 0) 0x7f50d40a7040 con 0x7f50d40ae8f0 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50da7c1700 1 --2- 172.21.15.115:0/143445666 >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d97bf700 1 --2- 172.21.15.115:0/143445666 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50c40098e0 tx=0x7f50c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef4989634c9368d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 1 -- 172.21.15.115:0/143445666 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400b640 con 0x7f50d40aecc0 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: dump: 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:00.842 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: _finish_auth 0 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 5 monclient: authenticate success, global_id 4418 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 1 -- 172.21.15.115:0/143445666 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c400b7a0 con 0x7f50d40aecc0 2023-09-30T01:05:00.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 4 set_mon_vals no callback set 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 1 -- 172.21.15.115:0/143445666 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400bb40 con 0x7f50d40aecc0 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50d8fbe700 10 monclient: dump: 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.844 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 10 monclient: shutdown 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 1 -- 172.21.15.115:0/143445666 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 msgr2=0x7f50d40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.830+0000 7f50dc224700 1 --2- 172.21.15.115:0/143445666 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50c40098e0 tx=0x7f50c40091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 -- 172.21.15.115:0/143445666 shutdown_connections 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 --2- 172.21.15.115:0/143445666 >> v2:172.21.15.115:3300/0 conn(0x7f50d41415c0 0x7f50d40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 --2- 172.21.15.115:0/143445666 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d40a5d10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 -- 172.21.15.115:0/143445666 >> 172.21.15.115:0/143445666 conn(0x7f50d406a460 msgr2=0x7f50d40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 -- 172.21.15.115:0/143445666 shutdown_connections 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 -- 172.21.15.115:0/143445666 wait complete. 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 10 monclient: build_initial_monmap 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 10 monclient: monmap: 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:00.846 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 librados: starting msgr at 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.831+0000 7f50dc224700 1 librados: starting objecter 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 Processor -- start 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 -- start start 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 librados: setting wanted keys 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 librados: calling monclient init 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient: init 2023-09-30T01:05:00.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-a con 0x7f50d40ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d414e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-c con 0x7f50d40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50d41415c0 0x7f50d414e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): picked mon.noname-b con 0x7f50d41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d414df20 con 0x7f50d40ae8f0 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40a7af0 con 0x7f50d41415c0 2023-09-30T01:05:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d413d6a0 con 0x7f50d40aecc0 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50dc224700 10 monclient(hunting): authenticate will time out at 1162.282713s 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50486/0 (socket says 172.21.15.115:50486) 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d414e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 1 -- 172.21.15.115:0/2355653440 learned_addr learned my addr 172.21.15.115:0/2355653440 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): get_auth_request con 0x7f50d40ae8f0 auth_method 0 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:00.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): get_auth_request con 0x7f50d40aecc0 auth_method 0 2023-09-30T01:05:00.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:00.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:00.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:00.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d9fc0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.832+0000 7f50d97bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 -- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 msgr2=0x7f50d414e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d414e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50da7c1700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50d41415c0 0x7f50d414e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 -- 172.21.15.115:0/2355653440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50d41415c0 msgr2=0x7f50d414e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50d41415c0 0x7f50d414e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 10 monclient: _finish_hunting 0 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 monclient: found mon.noname-a 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 -- 172.21.15.115:0/2355653440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc009a20 con 0x7f50d40ae8f0 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d97bf700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d414e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50d9fc0700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f50d0001ce0 tx=0x7f50d0007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 client.?.objecter ms_handle_connect 0x7f50d40ae8f0 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:00.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0017070 con 0x7f50d40ae8f0 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: dump: 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 5 monclient: authenticate success, global_id 4560 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d0014b60 con 0x7f50d40ae8f0 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 10 monclient: _renew_subs 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c4009420 con 0x7f50d40ae8f0 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0005900 con 0x7f50d40ae8f0 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50c9ffb700 10 monclient: dump: 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:00.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50cbfff700 4 set_mon_vals no callback set 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 10 monclient: _renew_subs 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d40a9bc0 con 0x7f50d40ae8f0 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.833+0000 7f50dc224700 1 librados: init done 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.834+0000 7f50dc224700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.834+0000 7f50dc224700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.834+0000 7f50dc224700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.834+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d4072960 con 0x7f50d40ae8f0 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f50d0005cb0 con 0x7f50d40ae8f0 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 1 --2- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 0x7f50b806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:00.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f50d008c120 con 0x7f50d40ae8f0 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 10 client.4560.objecter ms_dispatch 0x7f50d4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 3 client.4560.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 3 client.4560.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50d97bf700 1 --2- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 0x7f50b806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50d97bf700 10 monclient: get_auth_request con 0x7f50b8068890 auth_method 0 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.835+0000 7f50c9ffb700 20 client.4560.objecter dump_active .. 0 homeless 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.836+0000 7f50c9ffb700 10 client.4560.objecter ms_handle_connect 0x7f50b8068890 2023-09-30T01:05:00.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.836+0000 7f50d97bf700 1 --2- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 0x7f50b806ad40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50c40093f0 tx=0x7f50c4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:00.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.840+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50d0059a60 con 0x7f50d40ae8f0 2023-09-30T01:05:00.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.840+0000 7f50c9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:00.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:00.840+0000 7f50c9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.061+0000 7f50dc224700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.061+0000 7f50dc224700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.061+0000 7f50dc224700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.061+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50d4141a40 con 0x7f50d40ae8f0 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.062+0000 7f50c9ffb700 1 -- 172.21.15.115:0/2355653440 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f50d005d080 con 0x7f50d40ae8f0 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.062+0000 7f50c9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:01.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.062+0000 7f50c9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:01.063 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:01.063 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:01.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.063+0000 7f50dc224700 10 librados: watch_flush enter 2023-09-30T01:05:01.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 10 librados: watch_flush exit 2023-09-30T01:05:01.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 20 client.4560.objecter shutdown clearing up homeless session... 2023-09-30T01:05:01.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 10 client.4560.objecter successfully canceled tick 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 msgr2=0x7f50b806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 0x7f50b806ad40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50c40093f0 tx=0x7f50c4017000 comp rx=0 tx=0).stop 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 10 monclient: shutdown 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 msgr2=0x7f50d40a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f50d0001ce0 tx=0x7f50d0007e80 comp rx=0 tx=0).stop 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 shutdown_connections 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> v2:172.21.15.115:6816/115635 conn(0x7f50b8068890 0x7f50b806ad40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50d41415c0 0x7f50d414e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50d40ae8f0 0x7f50d40a94c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:01.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 --2- 172.21.15.115:0/2355653440 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50d40aecc0 0x7f50d414e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:01.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 >> 172.21.15.115:0/2355653440 conn(0x7f50d406a460 msgr2=0x7f50d40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:01.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 shutdown_connections 2023-09-30T01:05:01.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 -- 172.21.15.115:0/2355653440 wait complete. 2023-09-30T01:05:01.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:01.064+0000 7f50dc224700 1 librados: shutdown 2023-09-30T01:05:04.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:04.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.204+0000 7f1d7a700700 10 monclient: get_monmap_and_config 2023-09-30T01:05:04.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.204+0000 7f1d7a700700 10 monclient: build_initial_monmap 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.204+0000 7f1d7a700700 10 monclient: monmap: 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:04.205599+0000 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:04.205599+0000 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:04.205 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:04.206 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 1 Processor -- start 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 1 -- start start 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 10 monclient: init 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1d740ae8f0 0x7f1d740a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-c con 0x7f1d740ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.205+0000 7f1d7a700700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-b con 0x7f1d740aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f1d741415c0 0x7f1d740adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-a con 0x7f1d741415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d740a7040 con 0x7f1d741415c0 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d740b19b0 con 0x7f1d740ae8f0 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740a57f0 con 0x7f1d740aecc0 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient(hunting): authenticate will time out at 1165.656043s 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f1d741415c0 0x7f1d740adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f1d741415c0 0x7f1d740adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37436/0 (socket says 172.21.15.115:37436) 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38242/0 (socket says 172.21.15.115:38242) 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 -- 172.21.15.115:0/2198750772 learned_addr learned my addr 172.21.15.115:0/2198750772 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): get_auth_request con 0x7f1d741415c0 auth_method 0 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): get_auth_request con 0x7f1d740aecc0 auth_method 0 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:04.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient(hunting): handle_auth_done global_id 4563 payload 274 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 -- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 msgr2=0x7f1d740b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 -- 172.21.15.115:0/2198750772 >> v2:172.21.15.157:3300/0 conn(0x7f1d740ae8f0 msgr2=0x7f1d740a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d73fff700 1 --2- 172.21.15.115:0/2198750772 >> v2:172.21.15.157:3300/0 conn(0x7f1d740ae8f0 0x7f1d740a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 --2- 172.21.15.115:0/2198750772 >> v2:172.21.15.157:3300/0 conn(0x7f1d740ae8f0 0x7f1d740a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient: _finish_hunting 0 2023-09-30T01:05:04.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 monclient: found mon.noname-a 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 -- 172.21.15.115:0/2198750772 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d740a5480 con 0x7f1d741415c0 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d737fe700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d78c9d700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d741415c0 0x7f1d740adb60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1d740b59c0 tx=0x7f1d6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8081dc89999a38a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 1 -- 172.21.15.115:0/2198750772 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00e070 con 0x7f1d741415c0 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: dump: 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:04.222 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: _finish_auth 0 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 1 -- 172.21.15.115:0/2198750772 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00ba20 con 0x7f1d741415c0 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 5 monclient: authenticate success, global_id 4563 2023-09-30T01:05:04.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 1 -- 172.21.15.115:0/2198750772 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c0092e0 con 0x7f1d741415c0 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d7a700700 4 set_mon_vals no callback set 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.206+0000 7f1d72ffd700 10 monclient: dump: 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.224 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 10 monclient: shutdown 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 -- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d741415c0 msgr2=0x7f1d740adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d741415c0 0x7f1d740adb60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1d740b59c0 tx=0x7f1d6c009840 comp rx=0 tx=0).stop 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 -- 172.21.15.115:0/2198750772 shutdown_connections 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d741415c0 0x7f1d740adb60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2198750772 >> v2:172.21.15.157:3300/0 conn(0x7f1d740ae8f0 0x7f1d740a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2198750772 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 -- 172.21.15.115:0/2198750772 >> 172.21.15.115:0/2198750772 conn(0x7f1d7406a460 msgr2=0x7f1d740630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 -- 172.21.15.115:0/2198750772 shutdown_connections 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.207+0000 7f1d7a700700 1 -- 172.21.15.115:0/2198750772 wait complete. 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.208+0000 7f1d7a700700 10 monclient: build_initial_monmap 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.208+0000 7f1d7a700700 10 monclient: monmap: 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:04.226 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.208+0000 7f1d7a700700 1 librados: starting msgr at 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.208+0000 7f1d7a700700 1 librados: starting objecter 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.208+0000 7f1d7a700700 1 Processor -- start 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 -- start start 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 librados: setting wanted keys 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 librados: calling monclient init 2023-09-30T01:05:04.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient: init 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-a con 0x7f1d740ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-c con 0x7f1d740aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 0x7f1d740a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): picked mon.noname-b con 0x7f1d741415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740ab490 con 0x7f1d740ae8f0 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740a9220 con 0x7f1d741415c0 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7413ecf0 con 0x7f1d740aecc0 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d7a700700 10 monclient(hunting): authenticate will time out at 1165.659758s 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37438/0 (socket says 172.21.15.115:37438) 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 1 -- 172.21.15.115:0/2083072273 learned_addr learned my addr 172.21.15.115:0/2083072273 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): get_auth_request con 0x7f1d740ae8f0 auth_method 0 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): get_auth_request con 0x7f1d740aecc0 auth_method 0 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.209+0000 7f1d737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d78c9d700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 0x7f1d740a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 10 monclient(hunting): handle_auth_done global_id 4566 payload 824 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 -- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 msgr2=0x7f1d740a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 -- 172.21.15.115:0/2083072273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 msgr2=0x7f1d740a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 0x7f1d740a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 10 monclient: _finish_hunting 0 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 monclient: found mon.noname-a 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 -- 172.21.15.115:0/2083072273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64009420 con 0x7f1d740ae8f0 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d78c9d700 10 monclient: get_auth_request con 0x7f1d741415c0 auth_method 0 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d737fe700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:04.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d78c9d700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 0x7f1d740a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d73fff700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800c7d0 tx=0x7f1d68008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 client.?.objecter ms_handle_connect 0x7f1d740ae8f0 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68011600 con 0x7f1d740ae8f0 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: dump: 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:04.232 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: _finish_auth 0 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 5 monclient: authenticate success, global_id 4566 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68011760 con 0x7f1d740ae8f0 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68011b20 con 0x7f1d740ae8f0 2023-09-30T01:05:04.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 10 monclient: _renew_subs 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c009ae0 con 0x7f1d740ae8f0 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d727fc700 4 set_mon_vals no callback set 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d5bfff700 10 monclient: dump: 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:04.234 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 10 monclient: _renew_subs 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d740a7730 con 0x7f1d740ae8f0 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.210+0000 7f1d7a700700 1 librados: init done 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.211+0000 7f1d7a700700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.211+0000 7f1d7a700700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:04.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.211+0000 7f1d7a700700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.211+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d740a9720 con 0x7f1d740ae8f0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.211+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1d68018350 con 0x7f1d740ae8f0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 1 --2- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 0x7f1d6006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d737fe700 1 --2- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 0x7f1d6006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1d68083dc0 con 0x7f1d740ae8f0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 10 client.4566.objecter ms_dispatch 0x7f1d74001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 3 client.4566.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 3 client.4566.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d737fe700 10 monclient: get_auth_request con 0x7f1d60068890 auth_method 0 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 20 client.4566.objecter dump_active .. 0 homeless 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d5bfff700 10 client.4566.objecter ms_handle_connect 0x7f1d60068890 2023-09-30T01:05:04.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.212+0000 7f1d737fe700 1 --2- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 0x7f1d6006ad40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d740a8910 tx=0x7f1d640091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:04.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.216+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d68051680 con 0x7f1d740ae8f0 2023-09-30T01:05:04.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.216+0000 7f1d5bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:04.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.216+0000 7f1d5bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:04.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d7a700700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:04.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d7a700700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d7a700700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d74072960 con 0x7f1d740ae8f0 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d5bfff700 1 -- 172.21.15.115:0/2083072273 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1d6801e080 con 0x7f1d740ae8f0 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d5bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.436+0000 7f1d5bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:04.437 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 10 librados: watch_flush enter 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 10 librados: watch_flush exit 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 20 client.4566.objecter shutdown clearing up homeless session... 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 10 client.4566.objecter successfully canceled tick 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 msgr2=0x7f1d6006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 0x7f1d6006ad40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d740a8910 tx=0x7f1d640091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 10 monclient: shutdown 2023-09-30T01:05:04.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 msgr2=0x7f1d7413e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800c7d0 tx=0x7f1d68008040 comp rx=0 tx=0).stop 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 shutdown_connections 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> v2:172.21.15.115:6816/115635 conn(0x7f1d60068890 0x7f1d6006ad40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d741415c0 0x7f1d740a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d740ae8f0 0x7f1d7413e3c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 --2- 172.21.15.115:0/2083072273 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f1d740aecc0 0x7f1d740a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 >> 172.21.15.115:0/2083072273 conn(0x7f1d7406a460 msgr2=0x7f1d74144710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 shutdown_connections 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 -- 172.21.15.115:0/2083072273 wait complete. 2023-09-30T01:05:04.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:04.438+0000 7f1d7a700700 1 librados: shutdown 2023-09-30T01:05:07.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.580+0000 7fd26f77a700 10 monclient: get_monmap_and_config 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.580+0000 7fd26f77a700 10 monclient: build_initial_monmap 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.580+0000 7fd26f77a700 10 monclient: monmap: 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:07.581807+0000 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:07.581807+0000 2023-09-30T01:05:07.581 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:07.582 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.581+0000 7fd26f77a700 1 Processor -- start 2023-09-30T01:05:07.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.581+0000 7fd26f77a700 1 -- start start 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.581+0000 7fd26f77a700 10 monclient: init 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd2680ae8f0 0x7fd2680a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-a con 0x7fd2680ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 0x7fd2680b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-c con 0x7fd2680aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2681415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680a7040 con 0x7fd2681415c0 2023-09-30T01:05:07.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680b19b0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680a57f0 con 0x7fd2680aecc0 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26f77a700 10 monclient(hunting): authenticate will time out at 1169.032352s 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd2680ae8f0 0x7fd2680a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26dd17700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd2681415c0 0x7fd2680adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:38116/0 (socket says 172.21.15.115:38116) 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26dd17700 1 -- 172.21.15.115:0/1468465363 learned_addr learned my addr 172.21.15.115:0/1468465363 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): get_auth_request con 0x7fd2680ae8f0 auth_method 0 2023-09-30T01:05:07.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26cd15700 1 --2- 172.21.15.115:0/1468465363 >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 0x7fd2680b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 -- 172.21.15.115:0/1468465363 >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 msgr2=0x7fd2680b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 --2- 172.21.15.115:0/1468465363 >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 0x7fd2680b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 -- 172.21.15.115:0/1468465363 >> v1:172.21.15.115:6789/0 conn(0x7fd2681415c0 legacy=0x7fd2680adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient: _finish_hunting 0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 monclient: found mon.noname-a 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd25ffff700 1 -- 172.21.15.115:0/1468465363 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684679527 0 0) 0x7fd2680a7040 con 0x7fd2681415c0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 -- 172.21.15.115:0/1468465363 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2680a5480 con 0x7fd2680ae8f0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26cd15700 10 monclient: get_auth_request con 0x7fd2680aecc0 auth_method 0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26cd15700 1 --2- 172.21.15.115:0/1468465363 >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 0x7fd2680b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd25ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.582+0000 7fd26d516700 1 --2- 172.21.15.115:0/1468465363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd2680a5d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd268064080 tx=0x7fd2580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fabe06bfba9553 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 1 -- 172.21.15.115:0/1468465363 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800b6b0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: dump: 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: _finish_auth 0 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 5 monclient: authenticate success, global_id 4569 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 1 -- 172.21.15.115:0/1468465363 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800b810 con 0x7fd2680ae8f0 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 4 set_mon_vals no callback set 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 1 -- 172.21.15.115:0/1468465363 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800bbb0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd25ffff700 10 monclient: dump: 2023-09-30T01:05:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 10 monclient: shutdown 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 -- 172.21.15.115:0/1468465363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 msgr2=0x7fd2680a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 --2- 172.21.15.115:0/1468465363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd2680a5d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd268064080 tx=0x7fd2580091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 -- 172.21.15.115:0/1468465363 shutdown_connections 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 --2- 172.21.15.115:0/1468465363 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd2680a5d10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 --2- 172.21.15.115:0/1468465363 >> v2:172.21.15.157:3300/0 conn(0x7fd2680aecc0 0x7fd2680b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 -- 172.21.15.115:0/1468465363 >> 172.21.15.115:0/1468465363 conn(0x7fd26806a460 msgr2=0x7fd2680630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 -- 172.21.15.115:0/1468465363 shutdown_connections 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.583+0000 7fd26f77a700 1 -- 172.21.15.115:0/1468465363 wait complete. 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 10 monclient: build_initial_monmap 2023-09-30T01:05:07.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 10 monclient: monmap: 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 librados: starting msgr at 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 librados: starting objecter 2023-09-30T01:05:07.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 Processor -- start 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 -- start start 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 librados: setting wanted keys 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 1 librados: calling monclient init 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.584+0000 7fd26f77a700 10 monclient: init 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-a con 0x7fd2680ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2680aecc0 0x7fd2680aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-b con 0x7fd2680aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:07.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 0x7fd26813ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): picked mon.noname-c con 0x7fd2681415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680aba60 con 0x7fd2680ae8f0 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680aa9f0 con 0x7fd2680aecc0 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680aa710 con 0x7fd2681415c0 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26f77a700 10 monclient(hunting): authenticate will time out at 1169.035355s 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37464/0 (socket says 172.21.15.115:37464) 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 -- 172.21.15.115:0/875704303 learned_addr learned my addr 172.21.15.115:0/875704303 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): get_auth_request con 0x7fd2680ae8f0 auth_method 0 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 0x7fd26813ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): get_auth_request con 0x7fd2681415c0 auth_method 0 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:07.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26cd15700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2680aecc0 0x7fd2680aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26cd15700 10 monclient(hunting): get_auth_request con 0x7fd2680aecc0 auth_method 0 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26cd15700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26cd15700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26cd15700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient(hunting): handle_auth_done global_id 4575 payload 824 2023-09-30T01:05:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 -- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 msgr2=0x7fd26813ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 0x7fd26813ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 -- 172.21.15.115:0/875704303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2680aecc0 msgr2=0x7fd2680aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2680aecc0 0x7fd2680aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient: _finish_hunting 0 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 monclient: found mon.noname-a 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 -- 172.21.15.115:0/875704303 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009420 con 0x7fd2680ae8f0 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26dd17700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 0x7fd26813ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd26d516700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd2680b0920 tx=0x7fd258017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd25d7fa700 10 client.?.objecter ms_handle_connect 0x7fd2680ae8f0 2023-09-30T01:05:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.585+0000 7fd25d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258011420 con 0x7fd2680ae8f0 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: dump: 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:07.605 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580115a0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 5 monclient: authenticate success, global_id 4575 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 10 monclient: _renew_subs 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2680a9ad0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258011960 con 0x7fd2680ae8f0 2023-09-30T01:05:07.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25d7fa700 10 monclient: dump: 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:07.607 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd25f7fe700 4 set_mon_vals no callback set 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 10 monclient: _renew_subs 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26813b250 con 0x7fd2680ae8f0 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.586+0000 7fd26f77a700 1 librados: init done 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.587+0000 7fd26f77a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.587+0000 7fd26f77a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.587+0000 7fd26f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.587+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2680aac30 con 0x7fd2680ae8f0 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.587+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd258011d50 con 0x7fd2680ae8f0 2023-09-30T01:05:07.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 1 --2- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 0x7fd24c06adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd26cd15700 1 --2- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 0x7fd24c06adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd258085f60 con 0x7fd2680ae8f0 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 10 client.4575.objecter ms_dispatch 0x7fd268001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 3 client.4575.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 3 client.4575.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd26cd15700 10 monclient: get_auth_request con 0x7fd24c068940 auth_method 0 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 20 client.4575.objecter dump_active .. 0 homeless 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd25d7fa700 10 client.4575.objecter ms_handle_connect 0x7fd24c068940 2023-09-30T01:05:07.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.588+0000 7fd26cd15700 1 --2- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 0x7fd24c06adf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd26813eed0 tx=0x7fd264006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:07.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.592+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2580b15a0 con 0x7fd2680ae8f0 2023-09-30T01:05:07.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.592+0000 7fd25d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:07.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.592+0000 7fd25d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:07.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.810+0000 7fd26f77a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:07.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd26f77a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:07.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd26f77a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:07.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd2680aac30 con 0x7fd2680ae8f0 2023-09-30T01:05:07.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd25d7fa700 1 -- 172.21.15.115:0/875704303 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd2680aac30 con 0x7fd2680ae8f0 2023-09-30T01:05:07.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd25d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:07.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.811+0000 7fd25d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:07.812 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:07.812 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:07.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 10 librados: watch_flush enter 2023-09-30T01:05:07.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 10 librados: watch_flush exit 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 20 client.4575.objecter shutdown clearing up homeless session... 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 10 client.4575.objecter successfully canceled tick 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 msgr2=0x7fd24c06adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 0x7fd24c06adf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd26813eed0 tx=0x7fd264006f90 comp rx=0 tx=0).stop 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 10 monclient: shutdown 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 msgr2=0x7fd26813e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd2680b0920 tx=0x7fd258017000 comp rx=0 tx=0).stop 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 shutdown_connections 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> v2:172.21.15.115:6816/115635 conn(0x7fd24c068940 0x7fd24c06adf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd2681415c0 0x7fd26813ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2680ae8f0 0x7fd26813e720 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 --2- 172.21.15.115:0/875704303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2680aecc0 0x7fd2680aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 >> 172.21.15.115:0/875704303 conn(0x7fd26806a460 msgr2=0x7fd268144710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 shutdown_connections 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 -- 172.21.15.115:0/875704303 wait complete. 2023-09-30T01:05:07.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:07.813+0000 7fd26f77a700 1 librados: shutdown 2023-09-30T01:05:10.826 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:10.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.953+0000 7fc806162700 10 monclient: get_monmap_and_config 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.953+0000 7fc806162700 10 monclient: build_initial_monmap 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.953+0000 7fc806162700 10 monclient: monmap: 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:10.954556+0000 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:10.954556+0000 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:10.954 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:10.955 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.954+0000 7fc806162700 1 Processor -- start 2023-09-30T01:05:10.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.954+0000 7fc806162700 1 -- start start 2023-09-30T01:05:10.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.954+0000 7fc806162700 10 monclient: init 2023-09-30T01:05:10.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.954+0000 7fc806162700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.954+0000 7fc806162700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8000af890 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8000afc60 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc80013cb00 0x7fc80013f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): picked mon.noname-c con 0x7fc80013cb00 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80006f740 con 0x7fc8000af890 2023-09-30T01:05:10.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80006daf0 con 0x7fc8000afc60 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80013f390 con 0x7fc80013cb00 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 10 monclient(hunting): authenticate will time out at 1172.405087s 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc8000af890 0x7fc80006d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:38132/0 (socket says 172.21.15.115:38132) 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 1 -- 172.21.15.115:0/7752480 learned_addr learned my addr 172.21.15.115:0/7752480 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 1 --2- 172.21.15.115:0/7752480 >> v2:172.21.15.157:3300/0 conn(0x7fc80013cb00 0x7fc80013f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870641081 0 0) 0x7fc80006f740 con 0x7fc8000af890 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): my global_id is 4578 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8003570 con 0x7fc8000af890 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): get_auth_request con 0x7fc80013cb00 auth_method 0 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635795433 0 0) 0x7fc7e8003570 con 0x7fc8000af890 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 >> v2:172.21.15.157:3300/0 conn(0x7fc80013cb00 msgr2=0x7fc80013f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 --2- 172.21.15.115:0/7752480 >> v2:172.21.15.157:3300/0 conn(0x7fc80013cb00 0x7fc80013f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 >> v1:172.21.15.157:6789/0 conn(0x7fc8000afc60 legacy=0x7fc80013c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80013be00 con 0x7fc8000af890 2023-09-30T01:05:10.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: _finish_auth 0 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7fc7f4001470 con 0x7fc8000af890 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 5 monclient: authenticate success, global_id 4578 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: dump: 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 >> v1:172.21.15.115:6789/0 conn(0x7fc8000af890 legacy=0x7fc80006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 0x7fc7e8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): picked mon.a con 0x7fc7e8003a60 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8004810 0x7fc7e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): picked mon.b con 0x7fc7e8004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7e80051c0 0x7fc7e800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): picked mon.c con 0x7fc7e80051c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80013f390 con 0x7fc7e8003a60 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e800c0e0 con 0x7fc7e8004810 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e800c2a0 con 0x7fc7e80051c0 2023-09-30T01:05:10.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fe7fc700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc806162700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 0x7fc7e8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 10 monclient(hunting): get_auth_request con 0x7fc7e8003a60 auth_method 0 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7e80051c0 0x7fc7e800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7ff7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): get_auth_request con 0x7fc7e80051c0 auth_method 0 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7fffff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.955+0000 7fc7feffd700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8004810 0x7fc7e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): get_auth_request con 0x7fc7e8004810 auth_method 0 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:10.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient(hunting): handle_auth_done global_id 4578 payload 293 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 -- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7e80051c0 msgr2=0x7fc7e800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7e80051c0 0x7fc7e800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 -- 172.21.15.115:0/7752480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8004810 msgr2=0x7fc7e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8004810 0x7fc7e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 monclient: found mon.a 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:10.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 -- 172.21.15.115:0/7752480 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e8003570 con 0x7fc7e8003a60 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 10 monclient: _finish_auth 0 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fffff700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7e80051c0 0x7fc7e800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fffff700 1 -- 172.21.15.115:0/7752480 reap_dead start 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 0x7fc7e8003e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400c3c0 tx=0x7fc7f4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96745f710a7e9b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f400cc90 con 0x7fc7e8003a60 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: dump: 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.977 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc806162700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f400ce10 con 0x7fc7e8003a60 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc806162700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:10.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc806162700 4 set_mon_vals no callback set 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 1 -- 172.21.15.115:0/7752480 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4005b70 con 0x7fc7e8003a60 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.956+0000 7fc7fe7fc700 10 monclient: dump: 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:10.979 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 10 monclient: shutdown 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 -- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 msgr2=0x7fc7e8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 0x7fc7e8003e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400c3c0 tx=0x7fc7f4001e70 comp rx=0 tx=0).stop 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 -- 172.21.15.115:0/7752480 shutdown_connections 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 --2- 172.21.15.115:0/7752480 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7e8003a60 0x7fc7e8003e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 -- 172.21.15.115:0/7752480 >> 172.21.15.115:0/7752480 conn(0x7fc8000634c0 msgr2=0x7fc8000638c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 -- 172.21.15.115:0/7752480 shutdown_connections 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 -- 172.21.15.115:0/7752480 wait complete. 2023-09-30T01:05:10.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 10 monclient: build_initial_monmap 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 10 monclient: monmap: 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 librados: starting msgr at 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.957+0000 7fc806162700 1 librados: starting objecter 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 Processor -- start 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 -- start start 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 librados: setting wanted keys 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 librados: calling monclient init 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient: init 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 0x7fc8000a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): picked mon.noname-b con 0x7fc7e8003a60 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): picked mon.noname-c con 0x7fc8001492d0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): picked mon.noname-a con 0x7fc800149c80 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000a8910 con 0x7fc800149c80 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000a8a80 con 0x7fc7e8003a60 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000a9e30 con 0x7fc8001492d0 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc806162700 10 monclient(hunting): authenticate will time out at 1172.408854s 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38280/0 (socket says 172.21.15.115:38280) 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 1 -- 172.21.15.115:0/2095652678 learned_addr learned my addr 172.21.15.115:0/2095652678 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37482/0 (socket says 172.21.15.115:37482) 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 10 monclient(hunting): get_auth_request con 0x7fc8001492d0 auth_method 0 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7feffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 10 monclient(hunting): get_auth_request con 0x7fc800149c80 auth_method 0 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.958+0000 7fc7fffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:10.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 0x7fc8000a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 10 monclient(hunting): get_auth_request con 0x7fc7e8003a60 auth_method 0 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:10.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 -- 172.21.15.115:0/2095652678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 msgr2=0x7fc8000a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 0x7fc8000a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 -- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 msgr2=0x7fc80014c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient: _finish_hunting 0 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 monclient: found mon.noname-c 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 -- 172.21.15.115:0/2095652678 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000a8d60 con 0x7fc8001492d0 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 0x7fc8000a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7fffff700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:10.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7feffd700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0002be0 tx=0x7fc7f0007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 client.?.objecter ms_handle_connect 0x7fc8001492d0 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000f940 con 0x7fc8001492d0 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: dump: 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: _finish_auth 0 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f000faa0 con 0x7fc8001492d0 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 5 monclient: authenticate success, global_id 4433 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 10 monclient: _renew_subs 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc800146b60 con 0x7fc8001492d0 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:10.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f000fe60 con 0x7fc8001492d0 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7e77fe700 10 monclient: dump: 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:10.989 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc7fdffb700 4 set_mon_vals no callback set 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 10 monclient: _renew_subs 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80014cb00 con 0x7fc8001492d0 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.959+0000 7fc806162700 1 librados: init done 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.960+0000 7fc806162700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.960+0000 7fc806162700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.960+0000 7fc806162700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.960+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80006e330 con 0x7fc8001492d0 2023-09-30T01:05:10.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.960+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc7f0013600 con 0x7fc8001492d0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 1 --2- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 0x7fc7ec06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 0x7fc7ec06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc7f00844e0 con 0x7fc8001492d0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 10 client.4433.objecter ms_dispatch 0x7fc800001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 3 client.4433.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 3 client.4433.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7ff7fe700 10 monclient: get_auth_request con 0x7fc7ec068890 auth_method 0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 20 client.4433.objecter dump_active .. 0 homeless 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7e77fe700 10 client.4433.objecter ms_handle_connect 0x7fc7ec068890 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.961+0000 7fc7ff7fe700 1 --2- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 0x7fc7ec06ad40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4007cf0 tx=0x7fc7f4015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.965+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7f0051da0 con 0x7fc8001492d0 2023-09-30T01:05:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.965+0000 7fc7e77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:10.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:10.965+0000 7fc7e77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:11.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc806162700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:11.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc806162700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:11.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc806162700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:11.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc8000a9f70 con 0x7fc8001492d0 2023-09-30T01:05:11.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc7e77fe700 1 -- 172.21.15.115:0/2095652678 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc8000a9f70 con 0x7fc8001492d0 2023-09-30T01:05:11.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc7e77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:11.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.189+0000 7fc7e77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:11.190 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:11.190 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 10 librados: watch_flush enter 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 10 librados: watch_flush exit 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 20 client.4433.objecter shutdown clearing up homeless session... 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 10 client.4433.objecter successfully canceled tick 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 msgr2=0x7fc7ec06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 0x7fc7ec06ad40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4007cf0 tx=0x7fc7f4015000 comp rx=0 tx=0).stop 2023-09-30T01:05:11.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 10 monclient: shutdown 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 msgr2=0x7fc800149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0002be0 tx=0x7fc7f0007d10 comp rx=0 tx=0).stop 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 shutdown_connections 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> v2:172.21.15.115:6816/115635 conn(0x7fc7ec068890 0x7fc7ec06ad40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc800149c80 0x7fc80014c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7e8003a60 0x7fc8000a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 --2- 172.21.15.115:0/2095652678 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc8001492d0 0x7fc800149740 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 >> 172.21.15.115:0/2095652678 conn(0x7fc8000634c0 msgr2=0x7fc800070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 shutdown_connections 2023-09-30T01:05:11.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 -- 172.21.15.115:0/2095652678 wait complete. 2023-09-30T01:05:11.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:11.191+0000 7fc806162700 1 librados: shutdown 2023-09-30T01:05:14.204 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:14.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.332+0000 7f8182411700 10 monclient: get_monmap_and_config 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.332+0000 7f8182411700 10 monclient: build_initial_monmap 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.332+0000 7f8182411700 10 monclient: monmap: 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:14.333574+0000 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:14.333574+0000 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:14.333 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:14.334 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:14.334 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.333+0000 7f8182411700 1 Processor -- start 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.333+0000 7f8182411700 1 -- start start 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.333+0000 7f8182411700 10 monclient: init 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.333+0000 7f8182411700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f817c0ae770 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c06f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:14.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): picked mon.noname-b con 0x7f817c06c7b0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f817c137f60 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c06e5f0 con 0x7f817c0ae770 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c06d070 con 0x7f817c137f60 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c138a90 con 0x7f817c06c7b0 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient(hunting): authenticate will time out at 1175.784118s 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817bfff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f817c0ae770 0x7f817c06ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:38852/0 (socket says 172.21.15.115:38852) 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817bfff700 1 -- 172.21.15.115:0/3732971285 learned_addr learned my addr 172.21.15.115:0/3732971285 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:14.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): get_auth_request con 0x7f817c06c7b0 auth_method 0 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient(hunting): handle_auth_done global_id 4436 payload 274 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 1 -- 172.21.15.115:0/3732971285 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066446709 0 0) 0x7f817c06e5f0 con 0x7f817c0ae770 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 -- 172.21.15.115:0/3732971285 >> v1:172.21.15.157:6789/0 conn(0x7f817c137f60 legacy=0x7f817c138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 -- 172.21.15.115:0/3732971285 >> v1:172.21.15.115:6789/0 conn(0x7f817c0ae770 legacy=0x7f817c06ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 -- 172.21.15.115:0/3732971285 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0aeb40 con 0x7f817c06c7b0 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817b7fe700 1 --2- 172.21.15.115:0/3732971285 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c06f350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f816c009910 tx=0x7f816c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b9f7efa5a3f2f7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 1 -- 172.21.15.115:0/3732971285 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c00b820 con 0x7f817c06c7b0 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: dump: 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:14.345 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: _finish_auth 0 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:14.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 5 monclient: authenticate success, global_id 4436 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 1 -- 172.21.15.115:0/3732971285 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c00b980 con 0x7f817c06c7b0 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f8182411700 4 set_mon_vals no callback set 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 1 -- 172.21.15.115:0/3732971285 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c00bd20 con 0x7f817c06c7b0 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.334+0000 7f817affd700 10 monclient: dump: 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.347 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 10 monclient: shutdown 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 -- 172.21.15.115:0/3732971285 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 msgr2=0x7f817c06f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 --2- 172.21.15.115:0/3732971285 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c06f350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f816c009910 tx=0x7f816c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 -- 172.21.15.115:0/3732971285 shutdown_connections 2023-09-30T01:05:14.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 --2- 172.21.15.115:0/3732971285 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c06f350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 -- 172.21.15.115:0/3732971285 >> 172.21.15.115:0/3732971285 conn(0x7f817c068900 msgr2=0x7f817c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 -- 172.21.15.115:0/3732971285 shutdown_connections 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.335+0000 7f8182411700 1 -- 172.21.15.115:0/3732971285 wait complete. 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 10 monclient: build_initial_monmap 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 10 monclient: monmap: 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.349 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 librados: starting msgr at 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 librados: starting objecter 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 Processor -- start 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 -- start start 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 librados: setting wanted keys 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 1 librados: calling monclient init 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.336+0000 7f8182411700 10 monclient: init 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:14.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): picked mon.noname-c con 0x7f817c06c7b0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 0x7f817c12f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): picked mon.noname-b con 0x7f817c0ae770 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 0x7f817c12fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): picked mon.noname-a con 0x7f817c137f60 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c13c9f0 con 0x7f817c137f60 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c13afc0 con 0x7f817c0ae770 2023-09-30T01:05:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c135430 con 0x7f817c06c7b0 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient(hunting): authenticate will time out at 1175.787215s 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34420/0 (socket says 172.21.15.115:34420) 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 0x7f817c12fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 1 -- 172.21.15.115:0/2701523593 learned_addr learned my addr 172.21.15.115:0/2701523593 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): get_auth_request con 0x7f817c06c7b0 auth_method 0 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:14.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): get_auth_request con 0x7f817c137f60 auth_method 0 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 0x7f817c12f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:14.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 10 monclient(hunting): get_auth_request con 0x7f817c0ae770 auth_method 0 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 -- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 msgr2=0x7f817c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 -- 172.21.15.115:0/2701523593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 msgr2=0x7f817c12f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 0x7f817c12f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient: _finish_hunting 0 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 monclient: found mon.noname-a 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:14.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 -- 172.21.15.115:0/2701523593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c009420 con 0x7f817c137f60 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817bfff700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817b7fe700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 0x7f817c12f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f81809ae700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 0x7f817c12fdc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f817400bad0 tx=0x7f8174009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 client.?.objecter ms_handle_connect 0x7f817c137f60 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400c8d0 con 0x7f817c137f60 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: dump: 2023-09-30T01:05:14.355 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: _finish_auth 0 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817400ca30 con 0x7f817c137f60 2023-09-30T01:05:14.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 5 monclient: authenticate success, global_id 4587 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient: _renew_subs 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c13b250 con 0x7f817c137f60 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400cdf0 con 0x7f817c137f60 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8163fff700 10 monclient: dump: 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.357 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f817a7fc700 4 set_mon_vals no callback set 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.337+0000 7f8182411700 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 10 monclient: _renew_subs 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:14.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c139f90 con 0x7f817c137f60 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 1 librados: init done 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.338+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c072960 con 0x7f817c137f60 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8174014620 con 0x7f817c137f60 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 1 --2- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 0x7f816806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8174083df0 con 0x7f817c137f60 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 10 client.4587.objecter ms_dispatch 0x7f817c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 3 client.4587.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:14.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 3 client.4587.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f817bfff700 1 --2- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 0x7f816806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f817bfff700 10 monclient: get_auth_request con 0x7f8168068890 auth_method 0 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.339+0000 7f8163fff700 20 client.4587.objecter dump_active .. 0 homeless 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.340+0000 7f8163fff700 10 client.4587.objecter ms_handle_connect 0x7f8168068890 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.340+0000 7f817bfff700 1 --2- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 0x7f816806ad40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8170007fd0 tx=0x7f8170007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.344+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81740516b0 con 0x7f817c137f60 2023-09-30T01:05:14.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.344+0000 7f8163fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:14.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.344+0000 7f8163fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:14.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8182411700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:14.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8182411700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:14.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8182411700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:14.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f817c06cd20 con 0x7f817c137f60 2023-09-30T01:05:14.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8163fff700 1 -- 172.21.15.115:0/2701523593 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8174054cd0 con 0x7f817c137f60 2023-09-30T01:05:14.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8163fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:14.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.566+0000 7f8163fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:14.567 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:14.567 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:14.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 10 librados: watch_flush enter 2023-09-30T01:05:14.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 10 librados: watch_flush exit 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 20 client.4587.objecter shutdown clearing up homeless session... 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 10 client.4587.objecter successfully canceled tick 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 msgr2=0x7f816806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 0x7f816806ad40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8170007fd0 tx=0x7f8170007df0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 10 monclient: shutdown 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 msgr2=0x7f817c12fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 0x7f817c12fdc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f817400bad0 tx=0x7f8174009760 comp rx=0 tx=0).stop 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 shutdown_connections 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> v2:172.21.15.115:6816/115635 conn(0x7f8168068890 0x7f816806ad40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f817c137f60 0x7f817c12fdc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f817c06c7b0 0x7f817c134080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 --2- 172.21.15.115:0/2701523593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f817c0ae770 0x7f817c12f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:14.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 >> 172.21.15.115:0/2701523593 conn(0x7f817c068900 msgr2=0x7f817c067600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:14.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 shutdown_connections 2023-09-30T01:05:14.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 -- 172.21.15.115:0/2701523593 wait complete. 2023-09-30T01:05:14.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:14.568+0000 7f8182411700 1 librados: shutdown 2023-09-30T01:05:17.581 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:17.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 10 monclient: get_monmap_and_config 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 10 monclient: build_initial_monmap 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 10 monclient: monmap: 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:17.710165+0000 2023-09-30T01:05:17.710 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:17.710165+0000 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 1 Processor -- start 2023-09-30T01:05:17.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 1 -- start start 2023-09-30T01:05:17.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.709+0000 7f2054b1b700 10 monclient: init 2023-09-30T01:05:17.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:17.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-b con 0x7f20500ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20500aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f20501415c0 0x7f20500adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-c con 0x7f20501415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500a7040 con 0x7f20500aecc0 2023-09-30T01:05:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20500a57f0 con 0x7f20501415c0 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500a5480 con 0x7f20500ae8f0 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f2054b1b700 10 monclient(hunting): authenticate will time out at 1179.160615s 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34442/0 (socket says 172.21.15.115:34442) 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 1 -- 172.21.15.115:0/3135002234 learned_addr learned my addr 172.21.15.115:0/3135002234 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): get_auth_request con 0x7f20500ae8f0 auth_method 0 2023-09-30T01:05:17.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:17.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:17.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:17.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:17.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.710+0000 7f204e59c700 10 monclient(hunting): handle_auth_done global_id 4442 payload 274 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 -- 172.21.15.115:0/3135002234 >> v2:172.21.15.157:3300/0 conn(0x7f20501415c0 msgr2=0x7f20500adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204ed9d700 1 --2- 172.21.15.115:0/3135002234 >> v2:172.21.15.157:3300/0 conn(0x7f20501415c0 0x7f20500adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 --2- 172.21.15.115:0/3135002234 >> v2:172.21.15.157:3300/0 conn(0x7f20501415c0 0x7f20500adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 -- 172.21.15.115:0/3135002234 >> v1:172.21.15.157:6789/0 conn(0x7f20500aecc0 legacy=0x7f20500b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:17.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 monclient: found mon.noname-b 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 -- 172.21.15.115:0/3135002234 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500a55c0 con 0x7f20500ae8f0 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204e59c700 1 --2- 172.21.15.115:0/3135002234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f20440099b0 tx=0x7f2044009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bdc2b5217d31f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 1 -- 172.21.15.115:0/3135002234 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400b820 con 0x7f20500ae8f0 2023-09-30T01:05:17.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: dump: 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.726 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: _finish_auth 0 2023-09-30T01:05:17.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 1 -- 172.21.15.115:0/3135002234 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204400b980 con 0x7f20500ae8f0 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 5 monclient: authenticate success, global_id 4442 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:17.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 1 -- 172.21.15.115:0/3135002234 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400bd20 con 0x7f20500ae8f0 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 4 set_mon_vals no callback set 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f204d59a700 10 monclient: dump: 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:17.729 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:17.730 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 10 monclient: shutdown 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 1 -- 172.21.15.115:0/3135002234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 msgr2=0x7f20500a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.711+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3135002234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f20440099b0 tx=0x7f2044009250 comp rx=0 tx=0).stop 2023-09-30T01:05:17.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 -- 172.21.15.115:0/3135002234 shutdown_connections 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3135002234 >> v2:172.21.15.157:3300/0 conn(0x7f20501415c0 0x7f20500adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3135002234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500ae8f0 0x7f20500a5d10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 -- 172.21.15.115:0/3135002234 >> 172.21.15.115:0/3135002234 conn(0x7f205006a460 msgr2=0x7f20500630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 -- 172.21.15.115:0/3135002234 shutdown_connections 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 -- 172.21.15.115:0/3135002234 wait complete. 2023-09-30T01:05:17.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 10 monclient: build_initial_monmap 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 10 monclient: monmap: 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:17.734 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 librados: starting msgr at 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 librados: starting objecter 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.712+0000 7f2054b1b700 1 Processor -- start 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 -- start start 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 librados: setting wanted keys 2023-09-30T01:05:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 librados: calling monclient init 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient: init 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20500ae8f0 0x7f20500a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-b con 0x7f20500ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-c con 0x7f20500aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 0x7f205013ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): picked mon.noname-a con 0x7f20501415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500aa290 con 0x7f20501415c0 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500aa570 con 0x7f20500ae8f0 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500aa850 con 0x7f20500aecc0 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f2054b1b700 10 monclient(hunting): authenticate will time out at 1179.163605s 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 0x7f205013ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34450/0 (socket says 172.21.15.115:34450) 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 1 -- 172.21.15.115:0/3848420434 learned_addr learned my addr 172.21.15.115:0/3848420434 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): get_auth_request con 0x7f20500aecc0 auth_method 0 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): get_auth_request con 0x7f20501415c0 auth_method 0 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:17.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204e59c700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20500ae8f0 0x7f20500a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204e59c700 10 monclient(hunting): get_auth_request con 0x7f20500ae8f0 auth_method 0 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.713+0000 7f204dd9b700 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2023-09-30T01:05:17.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 -- 172.21.15.115:0/3848420434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20500ae8f0 msgr2=0x7f20500a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20500ae8f0 0x7f20500a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 -- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 msgr2=0x7f205013ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 0x7f205013ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 monclient: found mon.noname-c 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 -- 172.21.15.115:0/3848420434 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440094f0 con 0x7f20500aecc0 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204ed9d700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 0x7f205013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:17.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204dd9b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f204000c5d0 tx=0x7f2040007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 client.?.objecter ms_handle_connect 0x7f20500aecc0 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040011590 con 0x7f20500aecc0 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: dump: 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:17.741 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20400116f0 con 0x7f20500aecc0 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 5 monclient: authenticate success, global_id 4445 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 10 monclient: _renew_subs 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500a97a0 con 0x7f20500aecc0 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040011ab0 con 0x7f20500aecc0 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f204cd99700 4 set_mon_vals no callback set 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f203e7fc700 10 monclient: dump: 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:17.743 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 10 monclient: _renew_subs 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205013f500 con 0x7f20500aecc0 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.714+0000 7f2054b1b700 1 librados: init done 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.715+0000 7f2054b1b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.715+0000 7f2054b1b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.715+0000 7f2054b1b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.715+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2050072960 con 0x7f20500aecc0 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.715+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f204001c460 con 0x7f20500aecc0 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 1 --2- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 0x7f203006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f204e59c700 1 --2- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 0x7f203006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:17.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2040083b70 con 0x7f20500aecc0 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 10 client.4445.objecter ms_dispatch 0x7f2050001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 3 client.4445.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 3 client.4445.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f204e59c700 10 monclient: get_auth_request con 0x7f2030068890 auth_method 0 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 20 client.4445.objecter dump_active .. 0 homeless 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f203e7fc700 10 client.4445.objecter ms_handle_connect 0x7f2030068890 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.716+0000 7f204e59c700 1 --2- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 0x7f203006ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20440099b0 tx=0x7f2044016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.720+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2040051430 con 0x7f20500aecc0 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.720+0000 7f203e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:17.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.720+0000 7f203e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.942+0000 7f2054b1b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.942+0000 7f2054b1b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.942+0000 7f2054b1b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.942+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f20500aa990 con 0x7f20500aecc0 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.943+0000 7f203e7fc700 1 -- 172.21.15.115:0/3848420434 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f20500aa990 con 0x7f20500aecc0 2023-09-30T01:05:17.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.943+0000 7f203e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:17.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.943+0000 7f203e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:17.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:17.944 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 10 librados: watch_flush enter 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 10 librados: watch_flush exit 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 20 client.4445.objecter shutdown clearing up homeless session... 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 10 client.4445.objecter successfully canceled tick 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 msgr2=0x7f203006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 0x7f203006ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20440099b0 tx=0x7f2044016000 comp rx=0 tx=0).stop 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 10 monclient: shutdown 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:17.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 msgr2=0x7f205013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f204000c5d0 tx=0x7f2040007f60 comp rx=0 tx=0).stop 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 shutdown_connections 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> v2:172.21.15.115:6816/115635 conn(0x7f2030068890 0x7f203006ad40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f20501415c0 0x7f205013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20500ae8f0 0x7f20500a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 --2- 172.21.15.115:0/3848420434 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f20500aecc0 0x7f205013e840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 >> 172.21.15.115:0/3848420434 conn(0x7f205006a460 msgr2=0x7f2050140d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 shutdown_connections 2023-09-30T01:05:17.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 -- 172.21.15.115:0/3848420434 wait complete. 2023-09-30T01:05:17.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:17.945+0000 7f2054b1b700 1 librados: shutdown 2023-09-30T01:05:20.958 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.086+0000 7fd8f7a29700 10 monclient: get_monmap_and_config 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.086+0000 7fd8f7a29700 10 monclient: build_initial_monmap 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.086+0000 7fd8f7a29700 10 monclient: monmap: 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:21.087729+0000 2023-09-30T01:05:21.087 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:21.087729+0000 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.087+0000 7fd8f7a29700 1 Processor -- start 2023-09-30T01:05:21.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.087+0000 7fd8f7a29700 1 -- start start 2023-09-30T01:05:21.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.087+0000 7fd8f7a29700 10 monclient: init 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8f00ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd8f00aecc0 0x7fd8f00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-a con 0x7fd8f00aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8f01415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00a7040 con 0x7fd8f00ae8f0 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00a57f0 con 0x7fd8f01415c0 2023-09-30T01:05:21.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00a5480 con 0x7fd8f00aecc0 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient(hunting): authenticate will time out at 1182.538213s 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd8f00aecc0 0x7fd8f00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f57c5700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd8f00ae8f0 0x7fd8f00b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:38864/0 (socket says 172.21.15.115:38864) 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f57c5700 1 -- 172.21.15.115:0/2001315479 learned_addr learned my addr 172.21.15.115:0/2001315479 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): get_auth_request con 0x7fd8f00aecc0 auth_method 0 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:21.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2001315479 >> v1:172.21.15.157:6789/0 conn(0x7fd8f01415c0 legacy=0x7fd8f00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2001315479 >> v1:172.21.15.115:6789/0 conn(0x7fd8f00ae8f0 legacy=0x7fd8f00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient: _finish_hunting 0 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 monclient: found mon.noname-a 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2001315479 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00a55c0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 1 -- 172.21.15.115:0/2001315479 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274589221 0 0) 0x7fd8f00a7040 con 0x7fd8f00ae8f0 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f4fc4700 1 --2- 172.21.15.115:0/2001315479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a5d10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098e0 tx=0x7fd8e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9420b5b80e7412ac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 1 -- 172.21.15.115:0/2001315479 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0009e80 con 0x7fd8f00aecc0 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: dump: 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:21.101 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: _finish_auth 0 2023-09-30T01:05:21.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 5 monclient: authenticate success, global_id 4593 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 1 -- 172.21.15.115:0/2001315479 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000b860 con 0x7fd8f00aecc0 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 1 -- 172.21.15.115:0/2001315479 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000bbe0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8f7a29700 4 set_mon_vals no callback set 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.088+0000 7fd8e7fff700 10 monclient: dump: 2023-09-30T01:05:21.103 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 10 monclient: shutdown 2023-09-30T01:05:21.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2001315479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 msgr2=0x7fd8f00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2001315479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a5d10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098e0 tx=0x7fd8e00091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2001315479 shutdown_connections 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2001315479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a5d10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2001315479 >> 172.21.15.115:0/2001315479 conn(0x7fd8f006a460 msgr2=0x7fd8f00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2001315479 shutdown_connections 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.089+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2001315479 wait complete. 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 10 monclient: build_initial_monmap 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 10 monclient: monmap: 2023-09-30T01:05:21.105 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 librados: starting msgr at 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 librados: starting objecter 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 Processor -- start 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 -- start start 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 librados: setting wanted keys 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 1 librados: calling monclient init 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.090+0000 7fd8f7a29700 10 monclient: init 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 0x7fd8f013e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-b con 0x7fd8f00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-a con 0x7fd8f00aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8f01415c0 0x7fd8f00a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): picked mon.noname-c con 0x7fd8f01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00a8210 con 0x7fd8f00aecc0 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00aa2e0 con 0x7fd8f00ae8f0 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00ab6c0 con 0x7fd8f01415c0 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f7a29700 10 monclient(hunting): authenticate will time out at 1182.541510s 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 0x7fd8f013e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 0x7fd8f013e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.115:46606/0 (socket says 172.21.15.115:46606) 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 1 -- 172.21.15.115:0/2125197970 learned_addr learned my addr 172.21.15.115:0/2125197970 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): get_auth_request con 0x7fd8f00ae8f0 auth_method 0 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): get_auth_request con 0x7fd8f00aecc0 auth_method 0 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8f01415c0 0x7fd8f00a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): get_auth_request con 0x7fd8f01415c0 auth_method 0 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f5fc6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f57c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8f01415c0 msgr2=0x7fd8f00a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8f01415c0 0x7fd8f00a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2125197970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 msgr2=0x7fd8f013e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 0x7fd8f013e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient: _finish_hunting 0 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 monclient: found mon.noname-a 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.091+0000 7fd8f4fc4700 1 -- 172.21.15.115:0/2125197970 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0009420 con 0x7fd8f00aecc0 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f4fc4700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a7790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0016000 tx=0x7fd8e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 client.?.objecter ms_handle_connect 0x7fd8f00aecc0 2023-09-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000bbe0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: dump: 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:21.113 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: _finish_auth 0 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 5 monclient: authenticate success, global_id 4599 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0000d10 con 0x7fd8f00aecc0 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 10 monclient: _renew_subs 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f00ab930 con 0x7fd8f00aecc0 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0011650 con 0x7fd8f00aecc0 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e57fa700 10 monclient: dump: 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8e77fe700 4 set_mon_vals no callback set 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 10 monclient: _renew_subs 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f00a9820 con 0x7fd8f00aecc0 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.092+0000 7fd8f7a29700 1 librados: init done 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8f7a29700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8f7a29700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8f7a29700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f00a99b0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd8e00119c0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.093+0000 7fd8e57fa700 1 --2- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 0x7fd8d406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8f57c5700 1 --2- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 0x7fd8d406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd8e0085410 con 0x7fd8f00aecc0 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 10 client.4599.objecter ms_dispatch 0x7fd8f0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 3 client.4599.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 3 client.4599.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8f57c5700 10 monclient: get_auth_request con 0x7fd8d4068890 auth_method 0 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 20 client.4599.objecter dump_active .. 0 homeless 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8e57fa700 10 client.4599.objecter ms_handle_connect 0x7fd8d4068890 2023-09-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.094+0000 7fd8f57c5700 1 --2- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 0x7fd8d406ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec007160 tx=0x7fd8ec000fe0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.098+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8e0052cb0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.098+0000 7fd8e57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.098+0000 7fd8e57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8f7a29700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8f7a29700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8f7a29700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd8f0072960 con 0x7fd8f00aecc0 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8e57fa700 1 -- 172.21.15.115:0/2125197970 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd8e00562d0 con 0x7fd8f00aecc0 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8e57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.319+0000 7fd8e57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:21.321 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:21.321 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 10 librados: watch_flush enter 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 10 librados: watch_flush exit 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 20 client.4599.objecter shutdown clearing up homeless session... 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 10 client.4599.objecter successfully canceled tick 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 msgr2=0x7fd8d406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 0x7fd8d406ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec007160 tx=0x7fd8ec000fe0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 10 monclient: shutdown 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 msgr2=0x7fd8f00a7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a7790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0016000 tx=0x7fd8e0017000 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 shutdown_connections 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> v2:172.21.15.115:6816/115635 conn(0x7fd8d4068890 0x7fd8d406ad40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8f01415c0 0x7fd8f00a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8f00ae8f0 0x7fd8f013e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 --2- 172.21.15.115:0/2125197970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8f00aecc0 0x7fd8f00a7790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:21.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.321+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 >> 172.21.15.115:0/2125197970 conn(0x7fd8f006a460 msgr2=0x7fd8f0144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:21.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.322+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 shutdown_connections 2023-09-30T01:05:21.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.322+0000 7fd8f7a29700 1 -- 172.21.15.115:0/2125197970 wait complete. 2023-09-30T01:05:21.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:21.322+0000 7fd8f7a29700 1 librados: shutdown 2023-09-30T01:05:24.334 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:24.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.462+0000 7f0b996f5700 10 monclient: get_monmap_and_config 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.462+0000 7f0b996f5700 10 monclient: build_initial_monmap 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.462+0000 7f0b996f5700 10 monclient: monmap: 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:24.463328+0000 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:24.463328+0000 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:24.463 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.462+0000 7f0b996f5700 1 Processor -- start 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 -- start start 2023-09-30T01:05:24.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient: init 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b940ae8f0 0x7f0b941415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-c con 0x7f0b940ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-b con 0x7f0b940aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0b940a5c80 0x7f0b940adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-a con 0x7f0b940a5c80 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b940a7040 con 0x7f0b940a5c80 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b940a5480 con 0x7f0b940ae8f0 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940a5760 con 0x7f0b940aecc0 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b996f5700 10 monclient(hunting): authenticate will time out at 1185.913695s 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0b940a5c80 0x7f0b940adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0b940a5c80 0x7f0b940adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47824/0 (socket says 172.21.15.115:47824) 2023-09-30T01:05:24.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47076/0 (socket says 172.21.15.115:47076) 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 1 -- 172.21.15.115:0/1607462429 learned_addr learned my addr 172.21.15.115:0/1607462429 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): get_auth_request con 0x7f0b940a5c80 auth_method 0 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): get_auth_request con 0x7f0b940aecc0 auth_method 0 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:24.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.463+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b92ffd700 1 --2- 172.21.15.115:0/1607462429 >> v2:172.21.15.157:3300/0 conn(0x7f0b940ae8f0 0x7f0b941415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b92ffd700 10 monclient(hunting): get_auth_request con 0x7f0b940ae8f0 auth_method 0 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 10 monclient(hunting): handle_auth_done global_id 4602 payload 274 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 -- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 msgr2=0x7f0b940b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 -- 172.21.15.115:0/1607462429 >> v2:172.21.15.157:3300/0 conn(0x7f0b940ae8f0 msgr2=0x7f0b941415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 --2- 172.21.15.115:0/1607462429 >> v2:172.21.15.157:3300/0 conn(0x7f0b940ae8f0 0x7f0b941415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 10 monclient: _finish_hunting 0 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 monclient: found mon.noname-a 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 -- 172.21.15.115:0/1607462429 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94141b00 con 0x7f0b940a5c80 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b927fc700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b937fe700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940a5c80 0x7f0b940adb60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0b940b1d50 tx=0x7f0b8c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef6dac1254e5cfa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 1 -- 172.21.15.115:0/1607462429 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00e070 con 0x7f0b940a5c80 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:24.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: dump: 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 1 -- 172.21.15.115:0/1607462429 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c00ba20 con 0x7f0b940a5c80 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b996f5700 5 monclient: authenticate success, global_id 4602 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b996f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b996f5700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 1 -- 172.21.15.115:0/1607462429 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0092e0 con 0x7f0b940a5c80 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b996f5700 4 set_mon_vals no callback set 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b91ffb700 10 monclient: dump: 2023-09-30T01:05:24.478 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.464+0000 7f0b996f5700 10 monclient: shutdown 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:24.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 -- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940a5c80 msgr2=0x7f0b940adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940a5c80 0x7f0b940adb60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0b940b1d50 tx=0x7f0b8c009840 comp rx=0 tx=0).stop 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 -- 172.21.15.115:0/1607462429 shutdown_connections 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940a5c80 0x7f0b940adb60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 --2- 172.21.15.115:0/1607462429 >> v2:172.21.15.157:3300/0 conn(0x7f0b940ae8f0 0x7f0b941415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 --2- 172.21.15.115:0/1607462429 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940aecc0 0x7f0b940b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 -- 172.21.15.115:0/1607462429 >> 172.21.15.115:0/1607462429 conn(0x7f0b9406a490 msgr2=0x7f0b940630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 -- 172.21.15.115:0/1607462429 shutdown_connections 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 1 -- 172.21.15.115:0/1607462429 wait complete. 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 10 monclient: build_initial_monmap 2023-09-30T01:05:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.465+0000 7f0b996f5700 10 monclient: monmap: 2023-09-30T01:05:24.481 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:24.481 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:24.481 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:24.481 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:24.481 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:24.482 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.482 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:24.482 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:24.482 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:24.482 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 librados: starting msgr at 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 librados: starting objecter 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 Processor -- start 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 -- start start 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 librados: setting wanted keys 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 librados: calling monclient init 2023-09-30T01:05:24.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient: init 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 0x7f0b9413e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-c con 0x7f0b940a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b940ae8f0 0x7f0b940ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-b con 0x7f0b940ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): picked mon.noname-a con 0x7f0b940aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940aadf0 con 0x7f0b940aecc0 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b941517c0 con 0x7f0b940ae8f0 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:24.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94152190 con 0x7f0b940a5c80 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b996f5700 10 monclient(hunting): authenticate will time out at 1185.916764s 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 0x7f0b9413e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47830/0 (socket says 172.21.15.115:47830) 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 1 -- 172.21.15.115:0/3330052759 learned_addr learned my addr 172.21.15.115:0/3330052759 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): get_auth_request con 0x7f0b940aecc0 auth_method 0 2023-09-30T01:05:24.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 0x7f0b9413e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47092/0 (socket says 172.21.15.115:47092) 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): get_auth_request con 0x7f0b940a5c80 auth_method 0 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b927fc700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b940ae8f0 0x7f0b940ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b927fc700 10 monclient(hunting): get_auth_request con 0x7f0b940ae8f0 auth_method 0 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.466+0000 7f0b92ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 -- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 msgr2=0x7f0b9413e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:24.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 0x7f0b9413e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 -- 172.21.15.115:0/3330052759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b940ae8f0 msgr2=0x7f0b940ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b940ae8f0 0x7f0b940ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 10 monclient: _finish_hunting 0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 monclient: found mon.noname-a 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 -- 172.21.15.115:0/3330052759 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84009420 con 0x7f0b940aecc0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b937fe700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b940b1d80 tx=0x7f0b8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 client.?.objecter ms_handle_connect 0x7f0b940aecc0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0032f0 con 0x7f0b940aecc0 2023-09-30T01:05:24.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: dump: 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:24.490 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: _finish_auth 0 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c008e50 con 0x7f0b940aecc0 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 5 monclient: authenticate success, global_id 4605 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 10 monclient: _renew_subs 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c009ae0 con 0x7f0b940aecc0 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0256b0 con 0x7f0b940aecc0 2023-09-30T01:05:24.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b7affd700 10 monclient: dump: 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:24.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 10 monclient: _renew_subs 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9414e800 con 0x7f0b940aecc0 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b917fa700 4 set_mon_vals no callback set 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.467+0000 7f0b996f5700 1 librados: init done 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.468+0000 7f0b996f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.468+0000 7f0b996f5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.468+0000 7f0b996f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.468+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b94151930 con 0x7f0b940aecc0 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.468+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0b8c003040 con 0x7f0b940aecc0 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 1 --2- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 0x7f0b8006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:24.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0b8c085870 con 0x7f0b940aecc0 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 10 client.4605.objecter ms_dispatch 0x7f0b94001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 3 client.4605.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 3 client.4605.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b92ffd700 1 --2- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 0x7f0b8006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b92ffd700 10 monclient: get_auth_request con 0x7f0b80068890 auth_method 0 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 20 client.4605.objecter dump_active .. 0 homeless 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b7affd700 10 client.4605.objecter ms_handle_connect 0x7f0b80068890 2023-09-30T01:05:24.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.469+0000 7f0b92ffd700 1 --2- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 0x7f0b8006ad40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b88006fd0 tx=0x7f0b88008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:24.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.473+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b8c0531e0 con 0x7f0b940aecc0 2023-09-30T01:05:24.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.473+0000 7f0b7affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:24.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.473+0000 7f0b7affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b996f5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b996f5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b996f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0b94152350 con 0x7f0b940aecc0 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b7affd700 1 -- 172.21.15.115:0/3330052759 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0b94152350 con 0x7f0b940aecc0 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b7affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.695+0000 7f0b7affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:24.696 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:24.697 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:24.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 10 librados: watch_flush enter 2023-09-30T01:05:24.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 10 librados: watch_flush exit 2023-09-30T01:05:24.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 20 client.4605.objecter shutdown clearing up homeless session... 2023-09-30T01:05:24.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 10 client.4605.objecter successfully canceled tick 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 msgr2=0x7f0b8006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 0x7f0b8006ad40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b88006fd0 tx=0x7f0b88008040 comp rx=0 tx=0).stop 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 10 monclient: shutdown 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 msgr2=0x7f0b94151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b940b1d80 tx=0x7f0b8c018000 comp rx=0 tx=0).stop 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 shutdown_connections 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> v2:172.21.15.115:6816/115635 conn(0x7f0b80068890 0x7f0b8006ad40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0b940aecc0 0x7f0b94151c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0b940a5c80 0x7f0b9413e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 --2- 172.21.15.115:0/3330052759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b940ae8f0 0x7f0b940ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:24.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 >> 172.21.15.115:0/3330052759 conn(0x7f0b9406a490 msgr2=0x7f0b940adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:24.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 shutdown_connections 2023-09-30T01:05:24.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 -- 172.21.15.115:0/3330052759 wait complete. 2023-09-30T01:05:24.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:24.697+0000 7f0b996f5700 1 librados: shutdown 2023-09-30T01:05:27.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:27.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.838+0000 7f2e254bc700 10 monclient: get_monmap_and_config 2023-09-30T01:05:27.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.838+0000 7f2e254bc700 10 monclient: build_initial_monmap 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.838+0000 7f2e254bc700 10 monclient: monmap: 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:27.839336+0000 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:27.839336+0000 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:27.839 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:27.840 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:27.840 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:27.840 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.838+0000 7f2e254bc700 1 Processor -- start 2023-09-30T01:05:27.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 -- start start 2023-09-30T01:05:27.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient: init 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2e200ae8f0 0x7f2e201415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-c con 0x7f2e200ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 0x7f2e200b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-b con 0x7f2e200aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e200a5c80 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e200a7040 con 0x7f2e200a5c80 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e200a5480 con 0x7f2e200ae8f0 2023-09-30T01:05:27.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e200a5760 con 0x7f2e200aecc0 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e254bc700 10 monclient(hunting): authenticate will time out at 1189.289699s 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1f7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f2e200a5c80 0x7f2e200adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:40530/0 (socket says 172.21.15.115:40530) 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 0x7f2e200b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1f7fe700 1 -- 172.21.15.115:0/3188660619 learned_addr learned my addr 172.21.15.115:0/3188660619 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): get_auth_request con 0x7f2e200aecc0 auth_method 0 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:27.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.839+0000 7f2e1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 1 --2- 172.21.15.115:0/3188660619 >> v2:172.21.15.157:3300/0 conn(0x7f2e200ae8f0 0x7f2e201415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): get_auth_request con 0x7f2e200ae8f0 auth_method 0 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 1 -- 172.21.15.115:0/3188660619 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872370500 0 0) 0x7f2e200a7040 con 0x7f2e200a5c80 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient(hunting): my global_id is 4608 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 1 -- 172.21.15.115:0/3188660619 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e08003570 con 0x7f2e200a5c80 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:27.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 -- 172.21.15.115:0/3188660619 >> v2:172.21.15.157:3300/0 conn(0x7f2e200ae8f0 msgr2=0x7f2e201415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 --2- 172.21.15.115:0/3188660619 >> v2:172.21.15.157:3300/0 conn(0x7f2e200ae8f0 0x7f2e201415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 -- 172.21.15.115:0/3188660619 >> v1:172.21.15.115:6789/0 conn(0x7f2e200a5c80 legacy=0x7f2e200adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 monclient: found mon.noname-b 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 -- 172.21.15.115:0/3188660619 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20141b00 con 0x7f2e200aecc0 2023-09-30T01:05:27.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1e7fc700 1 --2- 172.21.15.115:0/3188660619 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 0x7f2e200b1430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098e0 tx=0x7f2e100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e509e66845eab10 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 1 -- 172.21.15.115:0/3188660619 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e10011630 con 0x7f2e200aecc0 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: dump: 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:27.853 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e254bc700 5 monclient: authenticate success, global_id 4457 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e254bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 1 -- 172.21.15.115:0/3188660619 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e10011790 con 0x7f2e200aecc0 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e254bc700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:27.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 1 -- 172.21.15.115:0/3188660619 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e10011b30 con 0x7f2e200aecc0 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e254bc700 4 set_mon_vals no callback set 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.840+0000 7f2e1dffb700 10 monclient: dump: 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:27.855 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:27.857 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:27.857 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 10 monclient: shutdown 2023-09-30T01:05:27.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:27.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 -- 172.21.15.115:0/3188660619 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 msgr2=0x7f2e200b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 --2- 172.21.15.115:0/3188660619 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 0x7f2e200b1430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098e0 tx=0x7f2e100091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 -- 172.21.15.115:0/3188660619 shutdown_connections 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 --2- 172.21.15.115:0/3188660619 >> v2:172.21.15.157:3300/0 conn(0x7f2e200ae8f0 0x7f2e201415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 --2- 172.21.15.115:0/3188660619 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200aecc0 0x7f2e200b1430 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 -- 172.21.15.115:0/3188660619 >> 172.21.15.115:0/3188660619 conn(0x7f2e2006a490 msgr2=0x7f2e200630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 -- 172.21.15.115:0/3188660619 shutdown_connections 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 1 -- 172.21.15.115:0/3188660619 wait complete. 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 10 monclient: build_initial_monmap 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.841+0000 7f2e254bc700 10 monclient: monmap: 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:27.858 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 librados: starting msgr at 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 librados: starting objecter 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 Processor -- start 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 -- start start 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 librados: setting wanted keys 2023-09-30T01:05:27.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 librados: calling monclient init 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient: init 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-c con 0x7f2e200a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e200ae8f0 0x7f2e200a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-b con 0x7f2e200ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 0x7f2e2014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): picked mon.noname-a con 0x7f2e200aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e200a7a90 con 0x7f2e200aecc0 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:27.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e201514d0 con 0x7f2e200ae8f0 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e20151f30 con 0x7f2e200a5c80 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e254bc700 10 monclient(hunting): authenticate will time out at 1189.292925s 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47120/0 (socket says 172.21.15.115:47120) 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.842+0000 7f2e1effd700 1 -- 172.21.15.115:0/1507394143 learned_addr learned my addr 172.21.15.115:0/1507394143 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): get_auth_request con 0x7f2e200a5c80 auth_method 0 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 0x7f2e2014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): get_auth_request con 0x7f2e200aecc0 auth_method 0 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1e7fc700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e200ae8f0 0x7f2e200a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1e7fc700 10 monclient(hunting): get_auth_request con 0x7f2e200ae8f0 auth_method 0 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient(hunting): handle_auth_done global_id 4460 payload 824 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 -- 172.21.15.115:0/1507394143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e200ae8f0 msgr2=0x7f2e200a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e200ae8f0 0x7f2e200a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 -- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 msgr2=0x7f2e2014ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 0x7f2e2014ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient: _finish_hunting 0 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 monclient: found mon.noname-c 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:27.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 -- 172.21.15.115:0/1507394143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10009420 con 0x7f2e200a5c80 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1f7fe700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 0x7f2e2014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1effd700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400ec80 tx=0x7f2e140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 client.?.objecter ms_handle_connect 0x7f2e200a5c80 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400c850 con 0x7f2e200a5c80 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: dump: 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: _finish_auth 0 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1400c9b0 con 0x7f2e200a5c80 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 5 monclient: authenticate success, global_id 4460 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 10 monclient: _renew_subs 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:27.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e200a97f0 con 0x7f2e200a5c80 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400cd50 con 0x7f2e200a5c80 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e06ffd700 10 monclient: dump: 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:27.866 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 10 monclient: _renew_subs 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e1d7fa700 4 set_mon_vals no callback set 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e200a8820 con 0x7f2e200a5c80 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.843+0000 7f2e254bc700 1 librados: init done 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.844+0000 7f2e254bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.844+0000 7f2e254bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.844+0000 7f2e254bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:27.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.844+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e20072960 con 0x7f2e200a5c80 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.845+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2e14013680 con 0x7f2e200a5c80 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.845+0000 7f2e06ffd700 1 --2- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 0x7f2e0c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2e14084a00 con 0x7f2e200a5c80 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e1e7fc700 1 --2- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 0x7f2e0c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 10 client.4460.objecter ms_dispatch 0x7f2e20001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 3 client.4460.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 3 client.4460.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e1e7fc700 10 monclient: get_auth_request con 0x7f2e0c068890 auth_method 0 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 20 client.4460.objecter dump_active .. 0 homeless 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e06ffd700 10 client.4460.objecter ms_handle_connect 0x7f2e0c068890 2023-09-30T01:05:27.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.846+0000 7f2e1e7fc700 1 --2- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 0x7f2e0c06ad40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e100091e0 tx=0x7f2e10017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:27.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.849+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e140522a0 con 0x7f2e200a5c80 2023-09-30T01:05:27.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.849+0000 7f2e06ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:27.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:27.849+0000 7f2e06ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:28.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e254bc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:28.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e254bc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:28.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e254bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:28.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2e200b1970 con 0x7f2e200a5c80 2023-09-30T01:05:28.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e06ffd700 1 -- 172.21.15.115:0/1507394143 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2e140558c0 con 0x7f2e200a5c80 2023-09-30T01:05:28.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e06ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:28.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.073+0000 7f2e06ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:28.074 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:28.074 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 10 librados: watch_flush enter 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 10 librados: watch_flush exit 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 20 client.4460.objecter shutdown clearing up homeless session... 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 10 client.4460.objecter successfully canceled tick 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 msgr2=0x7f2e0c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:28.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 0x7f2e0c06ad40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e100091e0 tx=0x7f2e10017000 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 10 monclient: shutdown 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 msgr2=0x7f2e201519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400ec80 tx=0x7f2e140091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 shutdown_connections 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> v2:172.21.15.115:6816/115635 conn(0x7f2e0c068890 0x7f2e0c06ad40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e200aecc0 0x7f2e2014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2e200a5c80 0x7f2e201519f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 --2- 172.21.15.115:0/1507394143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e200ae8f0 0x7f2e200a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 >> 172.21.15.115:0/1507394143 conn(0x7f2e2006a490 msgr2=0x7f2e200adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 shutdown_connections 2023-09-30T01:05:28.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 -- 172.21.15.115:0/1507394143 wait complete. 2023-09-30T01:05:28.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:28.075+0000 7f2e254bc700 1 librados: shutdown 2023-09-30T01:05:31.088 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.216+0000 7ff1e3658700 10 monclient: get_monmap_and_config 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.216+0000 7ff1e3658700 10 monclient: build_initial_monmap 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.216+0000 7ff1e3658700 10 monclient: monmap: 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:31.217819+0000 2023-09-30T01:05:31.217 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:31.217819+0000 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.217+0000 7ff1e3658700 1 Processor -- start 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.217+0000 7ff1e3658700 1 -- start start 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.217+0000 7ff1e3658700 10 monclient: init 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 0x7ff1dc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-b con 0x7ff1dc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:31.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1dc0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1dc1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc0a7040 con 0x7ff1dc1415c0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc0a57f0 con 0x7ff1dc0aecc0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0a5480 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e3658700 10 monclient(hunting): authenticate will time out at 1192.668368s 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e1bf5700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff1dc1415c0 0x7ff1dc0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:40538/0 (socket says 172.21.15.115:40538) 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e1bf5700 1 -- 172.21.15.115:0/2114497786 learned_addr learned my addr 172.21.15.115:0/2114497786 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 --2- 172.21.15.115:0/2114497786 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 0x7ff1dc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): get_auth_request con 0x7ff1dc0ae8f0 auth_method 0 2023-09-30T01:05:31.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1cffff700 1 -- 172.21.15.115:0/2114497786 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341643505 0 0) 0x7ff1dc0a7040 con 0x7ff1dc1415c0 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1cffff700 10 monclient(hunting): my global_id is 4614 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1cffff700 1 -- 172.21.15.115:0/2114497786 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c8003570 con 0x7ff1dc1415c0 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 -- 172.21.15.115:0/2114497786 >> v1:172.21.15.157:6789/0 conn(0x7ff1dc0aecc0 legacy=0x7ff1dc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:31.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 -- 172.21.15.115:0/2114497786 >> v1:172.21.15.115:6789/0 conn(0x7ff1dc1415c0 legacy=0x7ff1dc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient: _finish_hunting 0 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 monclient: found mon.noname-b 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 -- 172.21.15.115:0/2114497786 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0a55c0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.218+0000 7ff1e13f4700 1 --2- 172.21.15.115:0/2114497786 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 0x7ff1dc0a5d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009910 tx=0x7ff1d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ed9ab1d8d92a24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 1 -- 172.21.15.115:0/2114497786 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000b680 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: dump: 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: _finish_auth 0 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 5 monclient: authenticate success, global_id 4463 2023-09-30T01:05:31.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 1 -- 172.21.15.115:0/2114497786 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d000b7e0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 1 -- 172.21.15.115:0/2114497786 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000bba0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 4 set_mon_vals no callback set 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1cffff700 10 monclient: dump: 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.232 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 10 monclient: shutdown 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 -- 172.21.15.115:0/2114497786 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 msgr2=0x7ff1dc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 --2- 172.21.15.115:0/2114497786 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 0x7ff1dc0a5d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009910 tx=0x7ff1d00091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 -- 172.21.15.115:0/2114497786 shutdown_connections 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 --2- 172.21.15.115:0/2114497786 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc0ae8f0 0x7ff1dc0a5d10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 -- 172.21.15.115:0/2114497786 >> 172.21.15.115:0/2114497786 conn(0x7ff1dc06a460 msgr2=0x7ff1dc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 -- 172.21.15.115:0/2114497786 shutdown_connections 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.219+0000 7ff1e3658700 1 -- 172.21.15.115:0/2114497786 wait complete. 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.220+0000 7ff1e3658700 10 monclient: build_initial_monmap 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.220+0000 7ff1e3658700 10 monclient: monmap: 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:31.234 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.220+0000 7ff1e3658700 1 librados: starting msgr at 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.220+0000 7ff1e3658700 1 librados: starting objecter 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.221+0000 7ff1e3658700 1 Processor -- start 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.221+0000 7ff1e3658700 1 -- start start 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.221+0000 7ff1e3658700 1 librados: setting wanted keys 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.221+0000 7ff1e3658700 1 librados: calling monclient init 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.221+0000 7ff1e3658700 10 monclient: init 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:05:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-a con 0x7ff1dc0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1dc0aecc0 0x7ff1dc0aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-b con 0x7ff1dc0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc1415c0 0x7ff1dc139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): picked mon.noname-c con 0x7ff1dc1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0aa990 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0abd30 con 0x7ff1dc0aecc0 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0a8a40 con 0x7ff1dc1415c0 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:31.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient(hunting): authenticate will time out at 1192.672144s 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc1415c0 0x7ff1dc139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47848/0 (socket says 172.21.15.115:47848) 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 -- 172.21.15.115:0/1297935795 learned_addr learned my addr 172.21.15.115:0/1297935795 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): get_auth_request con 0x7ff1dc0ae8f0 auth_method 0 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): get_auth_request con 0x7ff1dc1415c0 auth_method 0 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:31.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e1bf5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1dc0aecc0 0x7ff1dc0aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): get_auth_request con 0x7ff1dc0aecc0 auth_method 0 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:31.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e0bf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient(hunting): handle_auth_done global_id 4617 payload 824 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 -- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc1415c0 msgr2=0x7ff1dc139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc1415c0 0x7ff1dc139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 -- 172.21.15.115:0/1297935795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1dc0aecc0 msgr2=0x7ff1dc0aa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1dc0aecc0 0x7ff1dc0aa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient: _finish_hunting 0 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 monclient: found mon.noname-a 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:31.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 -- 172.21.15.115:0/1297935795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d0009420 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e13f4700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00091e0 tx=0x7ff1d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 client.?.objecter ms_handle_connect 0x7ff1dc0ae8f0 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000b950 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: dump: 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.240 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: _finish_auth 0 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d000bab0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 5 monclient: authenticate success, global_id 4617 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _renew_subs 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:31.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc13a2b0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d001a680 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cf7fe700 4 set_mon_vals no callback set 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1cd7fa700 10 monclient: dump: 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:31.242 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _renew_subs 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.222+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc14de60 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.223+0000 7ff1e3658700 1 librados: init done 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.223+0000 7ff1e3658700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.223+0000 7ff1e3658700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.223+0000 7ff1e3658700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:31.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.223+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc0abba0 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff1d001aa30 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 1 --2- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 0x7ff1c006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1e0bf3700 1 --2- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 0x7ff1c006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff1d0085a50 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 10 client.4617.objecter ms_dispatch 0x7ff1dc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 3 client.4617.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 3 client.4617.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1e0bf3700 10 monclient: get_auth_request con 0x7ff1c00688c0 auth_method 0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 20 client.4617.objecter dump_active .. 0 homeless 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1cd7fa700 10 client.4617.objecter ms_handle_connect 0x7ff1c00688c0 2023-09-30T01:05:31.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.224+0000 7ff1e0bf3700 1 --2- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 0x7ff1c006ad70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0b24d0 tx=0x7ff1d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:31.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.228+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1d0053310 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.228+0000 7ff1cd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:31.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.228+0000 7ff1cd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:31.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.453+0000 7ff1e3658700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:31.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.453+0000 7ff1e3658700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:31.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.453+0000 7ff1e3658700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:31.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.453+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1dc141180 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.454+0000 7ff1cd7fa700 1 -- 172.21.15.115:0/1297935795 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff1d0056930 con 0x7ff1dc0ae8f0 2023-09-30T01:05:31.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.454+0000 7ff1cd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:31.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.454+0000 7ff1cd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:31.455 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:31.456 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:31.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 10 librados: watch_flush enter 2023-09-30T01:05:31.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 10 librados: watch_flush exit 2023-09-30T01:05:31.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 20 client.4617.objecter shutdown clearing up homeless session... 2023-09-30T01:05:31.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 10 client.4617.objecter successfully canceled tick 2023-09-30T01:05:31.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 msgr2=0x7ff1c006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 0x7ff1c006ad70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0b24d0 tx=0x7ff1d8009040 comp rx=0 tx=0).stop 2023-09-30T01:05:31.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 10 monclient: shutdown 2023-09-30T01:05:31.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:31.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 msgr2=0x7ff1dc13ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:31.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00091e0 tx=0x7ff1d0017000 comp rx=0 tx=0).stop 2023-09-30T01:05:31.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 shutdown_connections 2023-09-30T01:05:31.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> v2:172.21.15.115:6816/115635 conn(0x7ff1c00688c0 0x7ff1c006ad70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff1dc1415c0 0x7ff1dc139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff1dc0ae8f0 0x7ff1dc13ebb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 --2- 172.21.15.115:0/1297935795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1dc0aecc0 0x7ff1dc0aa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:31.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 >> 172.21.15.115:0/1297935795 conn(0x7ff1dc06a460 msgr2=0x7ff1dc13fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:31.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 shutdown_connections 2023-09-30T01:05:31.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 -- 172.21.15.115:0/1297935795 wait complete. 2023-09-30T01:05:31.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:31.456+0000 7ff1e3658700 1 librados: shutdown 2023-09-30T01:05:34.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.599+0000 7fc7982be700 10 monclient: get_monmap_and_config 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.599+0000 7fc7982be700 10 monclient: build_initial_monmap 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.599+0000 7fc7982be700 10 monclient: monmap: 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:34.600569+0000 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:34.600569+0000 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.600 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 Processor -- start 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 -- start start 2023-09-30T01:05:34.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient: init 2023-09-30T01:05:34.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:34.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:05:34.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-b con 0x7fc7900ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 0x7fc7900b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-a con 0x7fc7900aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7901415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900a7040 con 0x7fc7901415c0 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7900b19b0 con 0x7fc7900aecc0 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900a57f0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.600+0000 7fc7982be700 10 monclient(hunting): authenticate will time out at 1196.051000s 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 0x7fc7900b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44832/0 (socket says 172.21.15.115:44832) 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 -- 172.21.15.115:0/4146124632 learned_addr learned my addr 172.21.15.115:0/4146124632 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): get_auth_request con 0x7fc7900ae8f0 auth_method 0 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): get_auth_request con 0x7fc7900aecc0 auth_method 0 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:34.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 -- 172.21.15.115:0/4146124632 >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 msgr2=0x7fc7900b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 --2- 172.21.15.115:0/4146124632 >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 0x7fc7900b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 -- 172.21.15.115:0/4146124632 >> v1:172.21.15.157:6789/0 conn(0x7fc7901415c0 legacy=0x7fc7900adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient: _finish_hunting 0 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 monclient: found mon.noname-b 2023-09-30T01:05:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 -- 172.21.15.115:0/4146124632 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900a5480 con 0x7fc7900ae8f0 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795859700 1 --2- 172.21.15.115:0/4146124632 >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 0x7fc7900b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc79605a700 1 --2- 172.21.15.115:0/4146124632 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0099b0 tx=0x7fc78c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccfbbadd11667da2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 1 -- 172.21.15.115:0/4146124632 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c009ea0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: dump: 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: _finish_auth 0 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 1 -- 172.21.15.115:0/4146124632 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c00b860 con 0x7fc7900ae8f0 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc7982be700 5 monclient: authenticate success, global_id 4469 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc7982be700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:34.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 1 -- 172.21.15.115:0/4146124632 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00bc00 con 0x7fc7900ae8f0 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc795058700 10 monclient: dump: 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:34.615 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc7982be700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.601+0000 7fc7982be700 4 set_mon_vals no callback set 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 10 monclient: shutdown 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 -- 172.21.15.115:0/4146124632 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 msgr2=0x7fc7900a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 --2- 172.21.15.115:0/4146124632 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0099b0 tx=0x7fc78c009250 comp rx=0 tx=0).stop 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 -- 172.21.15.115:0/4146124632 shutdown_connections 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 --2- 172.21.15.115:0/4146124632 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc7900a5d10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 --2- 172.21.15.115:0/4146124632 >> v2:172.21.15.115:3300/0 conn(0x7fc7900aecc0 0x7fc7900b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 -- 172.21.15.115:0/4146124632 >> 172.21.15.115:0/4146124632 conn(0x7fc79006a460 msgr2=0x7fc7900630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 -- 172.21.15.115:0/4146124632 shutdown_connections 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 1 -- 172.21.15.115:0/4146124632 wait complete. 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 10 monclient: build_initial_monmap 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.602+0000 7fc7982be700 10 monclient: monmap: 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:34.617 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 librados: starting msgr at 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 librados: starting objecter 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 Processor -- start 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 -- start start 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 librados: setting wanted keys 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 librados: calling monclient init 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient: init 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:05:34.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc79013e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-c con 0x7fc7900ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-a con 0x7fc7900aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7901415c0 0x7fc79014e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): picked mon.noname-b con 0x7fc7901415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900abae0 con 0x7fc7900aecc0 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900aa420 con 0x7fc7901415c0 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79014e8b0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.603+0000 7fc7982be700 10 monclient(hunting): authenticate will time out at 1196.053936s 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc79013e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60240/0 (socket says 172.21.15.115:60240) 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 1 -- 172.21.15.115:0/2483151687 learned_addr learned my addr 172.21.15.115:0/2483151687 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): get_auth_request con 0x7fc7900ae8f0 auth_method 0 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): get_auth_request con 0x7fc7900aecc0 auth_method 0 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:34.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79685b700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7901415c0 0x7fc79014e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79685b700 10 monclient(hunting): get_auth_request con 0x7fc7901415c0 auth_method 0 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79685b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79685b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:34.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79685b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 -- 172.21.15.115:0/2483151687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7901415c0 msgr2=0x7fc79014e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7901415c0 0x7fc79014e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 -- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 msgr2=0x7fc79014de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient: _finish_hunting 0 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 monclient: found mon.noname-c 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 -- 172.21.15.115:0/2483151687 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc780009420 con 0x7fc7900ae8f0 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc795859700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:34.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc79605a700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc79013e940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7900b0340 tx=0x7fc78c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 client.?.objecter ms_handle_connect 0x7fc7900ae8f0 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00bbe0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: dump: 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.623 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: _finish_auth 0 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 5 monclient: authenticate success, global_id 4472 2023-09-30T01:05:34.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 10 monclient: _renew_subs 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c011420 con 0x7fc7900ae8f0 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c0094f0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0117c0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7867fc700 10 monclient: dump: 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.626 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc794857700 4 set_mon_vals no callback set 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 10 monclient: _renew_subs 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7900a9360 con 0x7fc7900ae8f0 2023-09-30T01:05:34.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.604+0000 7fc7982be700 1 librados: init done 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.605+0000 7fc7982be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.605+0000 7fc7982be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.605+0000 7fc7982be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.605+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7900a9550 con 0x7fc7900ae8f0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc78c011bf0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 1 --2- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 0x7fc77406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc78c084ef0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 10 client.4472.objecter ms_dispatch 0x7fc790001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 3 client.4472.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 3 client.4472.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc795859700 1 --2- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 0x7fc77406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:34.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc795859700 10 monclient: get_auth_request con 0x7fc774068890 auth_method 0 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 20 client.4472.objecter dump_active .. 0 homeless 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc7867fc700 10 client.4472.objecter ms_handle_connect 0x7fc774068890 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.606+0000 7fc795859700 1 --2- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 0x7fc77406ad40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7900aae80 tx=0x7fc7800091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.610+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc78c0528e0 con 0x7fc7900ae8f0 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.610+0000 7fc7867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:34.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.610+0000 7fc7867fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:34.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7982be700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7982be700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7982be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc790072960 con 0x7fc7900ae8f0 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7867fc700 1 -- 172.21.15.115:0/2483151687 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc78c055f00 con 0x7fc7900ae8f0 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:34.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.839+0000 7fc7867fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:34.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:34.841 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:34.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 10 librados: watch_flush enter 2023-09-30T01:05:34.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 10 librados: watch_flush exit 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 20 client.4472.objecter shutdown clearing up homeless session... 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 10 client.4472.objecter successfully canceled tick 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 msgr2=0x7fc77406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 0x7fc77406ad40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7900aae80 tx=0x7fc7800091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 10 monclient: shutdown 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 msgr2=0x7fc79013e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc79013e940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7900b0340 tx=0x7fc78c017000 comp rx=0 tx=0).stop 2023-09-30T01:05:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 shutdown_connections 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> v2:172.21.15.115:6816/115635 conn(0x7fc774068890 0x7fc77406ad40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7901415c0 0x7fc79014e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc7900ae8f0 0x7fc79013e940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 --2- 172.21.15.115:0/2483151687 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc7900aecc0 0x7fc79014de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 >> 172.21.15.115:0/2483151687 conn(0x7fc79006a460 msgr2=0x7fc790140db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.841+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 shutdown_connections 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.842+0000 7fc7982be700 1 -- 172.21.15.115:0/2483151687 wait complete. 2023-09-30T01:05:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:34.842+0000 7fc7982be700 1 librados: shutdown 2023-09-30T01:05:37.854 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:37.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 10 monclient: get_monmap_and_config 2023-09-30T01:05:37.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 10 monclient: build_initial_monmap 2023-09-30T01:05:37.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 10 monclient: monmap: 2023-09-30T01:05:37.983 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:37.983 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:37.983 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:37.983120+0000 2023-09-30T01:05:37.983 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:37.983120+0000 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:37.984 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:37.985 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:37.985 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:37.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 1 Processor -- start 2023-09-30T01:05:37.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 1 -- start start 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.982+0000 7f97a6e75700 10 monclient: init 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97a00ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f97a00aecc0 0x7f97a00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:38.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-a con 0x7f97a00aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97a01415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a00a7040 con 0x7f97a01415c0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a00a57f0 con 0x7f97a00ae8f0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a00a5480 con 0x7f97a00aecc0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a6e75700 10 monclient(hunting): authenticate will time out at 1199.433625s 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f97a00aecc0 0x7f97a00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a5412700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f97a01415c0 0x7f97a00adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:42964/0 (socket says 172.21.15.115:42964) 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f97a5412700 1 -- 172.21.15.115:0/1272482733 learned_addr learned my addr 172.21.15.115:0/1272482733 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:38.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): get_auth_request con 0x7f97a00aecc0 auth_method 0 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.983+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 10 monclient(hunting): handle_auth_done global_id 4626 payload 274 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 1 -- 172.21.15.115:0/1272482733 >> v1:172.21.15.157:6789/0 conn(0x7f97a00ae8f0 legacy=0x7f97a00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 1 -- 172.21.15.115:0/1272482733 >> v1:172.21.15.115:6789/0 conn(0x7f97a01415c0 legacy=0x7f97a00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 1 monclient: found mon.noname-a 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 1 -- 172.21.15.115:0/1272482733 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a00a55c0 con 0x7f97a00aecc0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 1 -- 172.21.15.115:0/1272482733 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632857737 0 0) 0x7f97a00a7040 con 0x7f97a01415c0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979ffff700 1 --2- 172.21.15.115:0/1272482733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a00a5d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f978c0098e0 tx=0x7f978c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5af566dbe06fa7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 1 -- 172.21.15.115:0/1272482733 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c009e80 con 0x7f97a00aecc0 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:38.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: dump: 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:38.004 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 1 -- 172.21.15.115:0/1272482733 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c00b860 con 0x7f97a00aecc0 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 5 monclient: authenticate success, global_id 4626 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 1 -- 172.21.15.115:0/1272482733 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c00bbe0 con 0x7f97a00aecc0 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 4 set_mon_vals no callback set 2023-09-30T01:05:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f979f7fe700 10 monclient: dump: 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:38.006 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 10 monclient: shutdown 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 1 -- 172.21.15.115:0/1272482733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 msgr2=0x7f97a00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 1 --2- 172.21.15.115:0/1272482733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a00a5d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f978c0098e0 tx=0x7f978c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 1 -- 172.21.15.115:0/1272482733 shutdown_connections 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 1 --2- 172.21.15.115:0/1272482733 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a00a5d10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.984+0000 7f97a6e75700 1 -- 172.21.15.115:0/1272482733 >> 172.21.15.115:0/1272482733 conn(0x7f97a006a460 msgr2=0x7f97a00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 1 -- 172.21.15.115:0/1272482733 shutdown_connections 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 1 -- 172.21.15.115:0/1272482733 wait complete. 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 10 monclient: build_initial_monmap 2023-09-30T01:05:38.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 10 monclient: monmap: 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:38.008 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 1 librados: starting msgr at 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 1 librados: starting objecter 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.985+0000 7f97a6e75700 1 Processor -- start 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 -- start start 2023-09-30T01:05:38.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 librados: setting wanted keys 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 librados: calling monclient init 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient: init 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97a00ae8f0 0x7f97a013e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-b con 0x7f97a00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:38.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-a con 0x7f97a00aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f97a01415c0 0x7f97a006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): picked mon.noname-c con 0x7f97a01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a006e9d0 con 0x7f97a00aecc0 2023-09-30T01:05:38.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a006d800 con 0x7f97a00ae8f0 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a01521a0 con 0x7f97a01415c0 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f97a6e75700 10 monclient(hunting): authenticate will time out at 1199.436889s 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.986+0000 7f979ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:38.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60256/0 (socket says 172.21.15.115:60256) 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 -- 172.21.15.115:0/3507296610 learned_addr learned my addr 172.21.15.115:0/3507296610 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): get_auth_request con 0x7f97a00aecc0 auth_method 0 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:38.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f97a01415c0 0x7f97a006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:38.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): get_auth_request con 0x7f97a01415c0 auth_method 0 2023-09-30T01:05:38.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:38.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:38.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:38.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97a00ae8f0 0x7f97a013e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): get_auth_request con 0x7f97a00ae8f0 auth_method 0 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:38.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a5412700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:38.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a4c11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:38.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2023-09-30T01:05:38.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 -- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f97a01415c0 msgr2=0x7f97a006d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f97a01415c0 0x7f97a006d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 -- 172.21.15.115:0/3507296610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97a00ae8f0 msgr2=0x7f97a013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97a00ae8f0 0x7f97a013e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 monclient: found mon.noname-a 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 -- 172.21.15.115:0/3507296610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c009420 con 0x7f97a00aecc0 2023-09-30T01:05:38.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979ffff700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f97a00b0920 tx=0x7f978c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 client.?.objecter ms_handle_connect 0x7f97a00aecc0 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c01f070 con 0x7f97a00aecc0 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:38.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: dump: 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: _finish_auth 0 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c000c90 con 0x7f97a00aecc0 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 5 monclient: authenticate success, global_id 4632 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 10 monclient: _renew_subs 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a0152420 con 0x7f97a00aecc0 2023-09-30T01:05:38.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c025420 con 0x7f97a00aecc0 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979cff9700 10 monclient: dump: 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:38.022 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f979effd700 4 set_mon_vals no callback set 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 10 monclient: _renew_subs 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a0152900 con 0x7f97a00aecc0 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.987+0000 7f97a6e75700 1 librados: init done 2023-09-30T01:05:38.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.988+0000 7f97a6e75700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.988+0000 7f97a6e75700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.988+0000 7f97a6e75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.988+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a006d990 con 0x7f97a00aecc0 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f978c023070 con 0x7f97a00aecc0 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 1 --2- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 0x7f978406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f978c0856e0 con 0x7f97a00aecc0 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 10 client.4632.objecter ms_dispatch 0x7f97a0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 3 client.4632.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 3 client.4632.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:38.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f97a4c11700 1 --2- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 0x7f978406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f97a4c11700 10 monclient: get_auth_request con 0x7f9784068890 auth_method 0 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.989+0000 7f979cff9700 20 client.4632.objecter dump_active .. 0 homeless 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.990+0000 7f979cff9700 10 client.4632.objecter ms_handle_connect 0x7f9784068890 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.990+0000 7f97a4c11700 1 --2- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 0x7f978406ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f97940039e0 tx=0x7f9794007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.993+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f978c053020 con 0x7f97a00aecc0 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.993+0000 7f979cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:38.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:37.993+0000 7f979cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:38.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f97a6e75700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:38.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f97a6e75700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:38.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f97a6e75700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f97a0072960 con 0x7f97a00aecc0 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f979cff9700 1 -- 172.21.15.115:0/3507296610 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f978c056640 con 0x7f97a00aecc0 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f979cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.215+0000 7f979cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:38.216 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:38.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 10 librados: watch_flush enter 2023-09-30T01:05:38.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 10 librados: watch_flush exit 2023-09-30T01:05:38.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 20 client.4632.objecter shutdown clearing up homeless session... 2023-09-30T01:05:38.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 10 client.4632.objecter successfully canceled tick 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 msgr2=0x7f978406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 0x7f978406ad40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f97940039e0 tx=0x7f9794007ec0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 10 monclient: shutdown 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 msgr2=0x7f97a006cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f97a00b0920 tx=0x7f978c017000 comp rx=0 tx=0).stop 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 shutdown_connections 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> v2:172.21.15.115:6816/115635 conn(0x7f9784068890 0x7f978406ad40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f97a01415c0 0x7f97a006d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97a00ae8f0 0x7f97a013e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 --2- 172.21.15.115:0/3507296610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f97a00aecc0 0x7f97a006cbd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 >> 172.21.15.115:0/3507296610 conn(0x7f97a006a460 msgr2=0x7f97a0144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.217+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 shutdown_connections 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.218+0000 7f97a6e75700 1 -- 172.21.15.115:0/3507296610 wait complete. 2023-09-30T01:05:38.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:38.218+0000 7f97a6e75700 1 librados: shutdown 2023-09-30T01:05:41.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:41.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.362+0000 7fa153b87700 10 monclient: get_monmap_and_config 2023-09-30T01:05:41.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.362+0000 7fa153b87700 10 monclient: build_initial_monmap 2023-09-30T01:05:41.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.362+0000 7fa153b87700 10 monclient: monmap: 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:41.363927+0000 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:41.363927+0000 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:41.364 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.363+0000 7fa153b87700 1 Processor -- start 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.363+0000 7fa153b87700 1 -- start start 2023-09-30T01:05:41.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.363+0000 7fa153b87700 10 monclient: init 2023-09-30T01:05:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T01:05:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 0x7fa14c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-c con 0x7fa14c0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-b con 0x7fa14c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa14c1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0a7040 con 0x7fa14c1415c0 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14c0b19b0 con 0x7fa14c0ae8f0 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0a57f0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa153b87700 10 monclient(hunting): authenticate will time out at 1202.814387s 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa152124700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa14c1415c0 0x7fa14c0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:50444/0 (socket says 172.21.15.115:50444) 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:33950/0 (socket says 172.21.15.115:33950) 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa152124700 1 -- 172.21.15.115:0/4231076265 learned_addr learned my addr 172.21.15.115:0/4231076265 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): get_auth_request con 0x7fa14c0aecc0 auth_method 0 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 1 --2- 172.21.15.115:0/4231076265 >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 0x7fa14c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 10 monclient(hunting): get_auth_request con 0x7fa14c0ae8f0 auth_method 0 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient(hunting): handle_auth_done global_id 4478 payload 274 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa150921700 1 -- 172.21.15.115:0/4231076265 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189120123 0 0) 0x7fa14c0a7040 con 0x7fa14c1415c0 2023-09-30T01:05:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 -- 172.21.15.115:0/4231076265 >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 msgr2=0x7fa14c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 --2- 172.21.15.115:0/4231076265 >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 0x7fa14c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 -- 172.21.15.115:0/4231076265 >> v1:172.21.15.115:6789/0 conn(0x7fa14c1415c0 legacy=0x7fa14c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient: _finish_hunting 0 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 monclient: found mon.noname-b 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151122700 1 -- 172.21.15.115:0/4231076265 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c0a5480 con 0x7fa14c0aecc0 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa151923700 1 --2- 172.21.15.115:0/4231076265 >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 0x7fa14c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.364+0000 7fa150921700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa151122700 1 --2- 172.21.15.115:0/4231076265 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0098e0 tx=0x7fa13c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef2b272a37adcd4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 1 -- 172.21.15.115:0/4231076265 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c009e80 con 0x7fa14c0aecc0 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: dump: 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:41.379 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: _finish_auth 0 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 5 monclient: authenticate success, global_id 4478 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 1 -- 172.21.15.115:0/4231076265 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c00b860 con 0x7fa14c0aecc0 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:41.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 4 set_mon_vals no callback set 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 1 -- 172.21.15.115:0/4231076265 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00bbe0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa150921700 10 monclient: dump: 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 10 monclient: shutdown 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 -- 172.21.15.115:0/4231076265 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 msgr2=0x7fa14c0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 --2- 172.21.15.115:0/4231076265 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0098e0 tx=0x7fa13c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 -- 172.21.15.115:0/4231076265 shutdown_connections 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 --2- 172.21.15.115:0/4231076265 >> v2:172.21.15.157:3300/0 conn(0x7fa14c0ae8f0 0x7fa14c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 --2- 172.21.15.115:0/4231076265 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c0aecc0 0x7fa14c0b1470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 -- 172.21.15.115:0/4231076265 >> 172.21.15.115:0/4231076265 conn(0x7fa14c06a460 msgr2=0x7fa14c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 -- 172.21.15.115:0/4231076265 shutdown_connections 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.365+0000 7fa153b87700 1 -- 172.21.15.115:0/4231076265 wait complete. 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 10 monclient: build_initial_monmap 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 10 monclient: monmap: 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 librados: starting msgr at 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 librados: starting objecter 2023-09-30T01:05:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 Processor -- start 2023-09-30T01:05:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 -- start start 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 librados: setting wanted keys 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 1 librados: calling monclient init 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.366+0000 7fa153b87700 10 monclient: init 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa14c0ae8f0 0x7fa14c13e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-b con 0x7fa14c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-a con 0x7fa14c0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): picked mon.noname-c con 0x7fa14c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0ab1d0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c151ee0 con 0x7fa14c0ae8f0 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0ab3d0 con 0x7fa14c1415c0 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa153b87700 10 monclient(hunting): authenticate will time out at 1202.817382s 2023-09-30T01:05:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54768/0 (socket says 172.21.15.115:54768) 2023-09-30T01:05:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 -- 172.21.15.115:0/2534603713 learned_addr learned my addr 172.21.15.115:0/2534603713 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): get_auth_request con 0x7fa14c0aecc0 auth_method 0 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:33956/0 (socket says 172.21.15.115:33956) 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): get_auth_request con 0x7fa14c1415c0 auth_method 0 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151923700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa14c0ae8f0 0x7fa14c13e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient(hunting): handle_auth_done global_id 4638 payload 824 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 -- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 msgr2=0x7fa14c0a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 -- 172.21.15.115:0/2534603713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa14c0ae8f0 msgr2=0x7fa14c13e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa14c0ae8f0 0x7fa14c13e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient: _finish_hunting 0 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 monclient: found mon.noname-a 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 -- 172.21.15.115:0/2534603713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c009420 con 0x7fa14c0aecc0 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa152124700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa151122700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0a5930 tx=0x7fa13c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa141ffb700 10 client.?.objecter ms_handle_connect 0x7fa14c0aecc0 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.367+0000 7fa141ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00bbe0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: dump: 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c011420 con 0x7fa14c0aecc0 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 5 monclient: authenticate success, global_id 4638 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 10 monclient: _renew_subs 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c151a80 con 0x7fa14c0aecc0 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0117c0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa141ffb700 10 monclient: dump: 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa143fff700 4 set_mon_vals no callback set 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 10 client.4638.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 10 monclient: _renew_subs 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c0a99a0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.368+0000 7fa153b87700 1 librados: init done 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.369+0000 7fa153b87700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.369+0000 7fa153b87700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.369+0000 7fa153b87700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.369+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c0ab510 con 0x7fa14c0aecc0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.369+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa13c011b70 con 0x7fa14c0aecc0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 1 --2- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 0x7fa13006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa151923700 1 --2- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 0x7fa13006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa13c0858f0 con 0x7fa14c0aecc0 2023-09-30T01:05:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 10 client.4638.objecter ms_dispatch 0x7fa14c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 3 client.4638.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 3 client.4638.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa151923700 10 monclient: get_auth_request con 0x7fa130068890 auth_method 0 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 20 client.4638.objecter dump_active .. 0 homeless 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa141ffb700 10 client.4638.objecter ms_handle_connect 0x7fa130068890 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.370+0000 7fa151923700 1 --2- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 0x7fa13006ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa148006f00 tx=0x7fa148008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.374+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa14c0ab510 con 0x7fa14c0aecc0 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.374+0000 7fa141ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.374+0000 7fa141ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa153b87700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa153b87700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa153b87700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa14c0ab510 con 0x7fa14c0aecc0 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa141ffb700 1 -- 172.21.15.115:0/2534603713 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa14c0ab510 con 0x7fa14c0aecc0 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa141ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:41.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.595+0000 7fa141ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:41.597 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:41.597 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:41.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.597+0000 7fa153b87700 10 librados: watch_flush enter 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.597+0000 7fa153b87700 10 librados: watch_flush exit 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.597+0000 7fa153b87700 20 client.4638.objecter shutdown clearing up homeless session... 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.597+0000 7fa153b87700 10 client.4638.objecter successfully canceled tick 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 msgr2=0x7fa13006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 0x7fa13006ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa148006f00 tx=0x7fa148008040 comp rx=0 tx=0).stop 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 10 monclient: shutdown 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 msgr2=0x7fa14c151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0a5930 tx=0x7fa13c017000 comp rx=0 tx=0).stop 2023-09-30T01:05:41.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 shutdown_connections 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> v2:172.21.15.115:6816/115635 conn(0x7fa130068890 0x7fa13006ad40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa14c1415c0 0x7fa14c0a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa14c0ae8f0 0x7fa14c13e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 --2- 172.21.15.115:0/2534603713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa14c0aecc0 0x7fa14c151540 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 >> 172.21.15.115:0/2534603713 conn(0x7fa14c06a460 msgr2=0x7fa14c1448b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 shutdown_connections 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 -- 172.21.15.115:0/2534603713 wait complete. 2023-09-30T01:05:41.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:41.598+0000 7fa153b87700 1 librados: shutdown 2023-09-30T01:05:44.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:44.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.738+0000 7f5a77580700 10 monclient: get_monmap_and_config 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.738+0000 7f5a77580700 10 monclient: build_initial_monmap 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.738+0000 7f5a77580700 10 monclient: monmap: 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:44.739831+0000 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:44.739831+0000 2023-09-30T01:05:44.739 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:44.740 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.739+0000 7f5a77580700 1 Processor -- start 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.739+0000 7f5a77580700 1 -- start start 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.739+0000 7f5a77580700 10 monclient: init 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5a700ae8f0 0x7f5a700a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-c con 0x7f5a700ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a700aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a701415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700a7040 con 0x7f5a700aecc0 2023-09-30T01:05:44.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700a57f0 con 0x7f5a701415c0 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a700a5480 con 0x7f5a700ae8f0 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a77580700 10 monclient(hunting): authenticate will time out at 1206.190385s 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a74b1b700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f5a700aecc0 0x7f5a700b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:50452/0 (socket says 172.21.15.115:50452) 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a74b1b700 1 -- 172.21.15.115:0/2396918944 learned_addr learned my addr 172.21.15.115:0/2396918944 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 1 --2- 172.21.15.115:0/2396918944 >> v2:172.21.15.157:3300/0 conn(0x7f5a700ae8f0 0x7f5a700a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): get_auth_request con 0x7f5a700ae8f0 auth_method 0 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118743070 0 0) 0x7f5a700a7040 con 0x7f5a700aecc0 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): my global_id is 4641 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c003570 con 0x7f5a700aecc0 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906386168 0 0) 0x7f5a700a57f0 con 0x7f5a701415c0 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 10 monclient(hunting): my global_id is 4570 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a700a7040 con 0x7f5a701415c0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034660133 0 0) 0x7f5a5c003570 con 0x7f5a700aecc0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 >> v2:172.21.15.157:3300/0 conn(0x7f5a700ae8f0 msgr2=0x7f5a700a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.740+0000 7f5a67fff700 1 --2- 172.21.15.115:0/2396918944 >> v2:172.21.15.157:3300/0 conn(0x7f5a700ae8f0 0x7f5a700a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 >> v1:172.21.15.157:6789/0 conn(0x7f5a701415c0 legacy=0x7f5a700adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: _finish_hunting 0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700a55c0 con 0x7f5a700aecc0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: _finish_auth 0 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:44.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f5a60001470 con 0x7f5a700aecc0 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 5 monclient: authenticate success, global_id 4641 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: dump: 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.761 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 >> v1:172.21.15.115:6789/0 conn(0x7f5a700aecc0 legacy=0x7f5a700b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c003bf0 0x7f5a5c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): picked mon.b con 0x7f5a5c003bf0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 0x7f5a5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): picked mon.c con 0x7f5a5c004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a5c0051c0 0x7f5a5c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): picked mon.a con 0x7f5a5c0051c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700a5480 con 0x7f5a5c0051c0 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c00c0b0 con 0x7f5a5c003bf0 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c00c270 con 0x7f5a5c004810 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:44.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 0x7f5a5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c003bf0 0x7f5a5c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): get_auth_request con 0x7f5a5c004810 auth_method 0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a5c0051c0 0x7f5a5c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): get_auth_request con 0x7f5a5c003bf0 auth_method 0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): get_auth_request con 0x7f5a5c0051c0 auth_method 0 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a7531c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_done global_id 4641 payload 293 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 -- 172.21.15.115:0/2396918944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c003bf0 msgr2=0x7f5a5c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c003bf0 0x7f5a5c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 -- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a5c0051c0 msgr2=0x7f5a5c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a5c0051c0 0x7f5a5c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient: _finish_hunting 0 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 monclient: found mon.c 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 -- 172.21.15.115:0/2396918944 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c003570 con 0x7f5a5c004810 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 10 monclient: _finish_auth 0 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a5c0051c0 0x7f5a5c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:44.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a75b1d700 1 -- 172.21.15.115:0/2396918944 reap_dead start 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 0x7f5a5c004c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c2d0 tx=0x7f5a60001e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf667d0f45a1e680 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000cbc0 con 0x7f5a5c004810 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: dump: 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.768 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000cd20 con 0x7f5a5c004810 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 1 -- 172.21.15.115:0/2396918944 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60005b70 con 0x7f5a5c004810 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:44.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a77580700 4 set_mon_vals no callback set 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.741+0000 7f5a67fff700 10 monclient: dump: 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:44.770 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 10 monclient: shutdown 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 -- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 msgr2=0x7f5a5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 0x7f5a5c004c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000c2d0 tx=0x7f5a60001e10 comp rx=0 tx=0).stop 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 -- 172.21.15.115:0/2396918944 shutdown_connections 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 --2- 172.21.15.115:0/2396918944 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a5c004810 0x7f5a5c004c80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 -- 172.21.15.115:0/2396918944 >> 172.21.15.115:0/2396918944 conn(0x7f5a7006a460 msgr2=0x7f5a700630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:44.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 -- 172.21.15.115:0/2396918944 shutdown_connections 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 1 -- 172.21.15.115:0/2396918944 wait complete. 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 10 monclient: build_initial_monmap 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.742+0000 7f5a77580700 10 monclient: monmap: 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.772 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 librados: starting msgr at 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 librados: starting objecter 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 Processor -- start 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 -- start start 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 librados: setting wanted keys 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 1 librados: calling monclient init 2023-09-30T01:05:44.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.743+0000 7f5a77580700 10 monclient: init 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c004810 0x7f5a7014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 0x7f5a700a90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-a con 0x7f5a7013d6d0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): picked mon.noname-c con 0x7f5a7014ea80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7014e060 con 0x7f5a7013d6d0 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700aadf0 con 0x7f5a5c004810 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700a9b60 con 0x7f5a7014ea80 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient(hunting): authenticate will time out at 1206.194115s 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 0x7f5a700a90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:33986/0 (socket says 172.21.15.115:33986) 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 1 -- 172.21.15.115:0/1384783921 learned_addr learned my addr 172.21.15.115:0/1384783921 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): get_auth_request con 0x7f5a7013d6d0 auth_method 0 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): get_auth_request con 0x7f5a7014ea80 auth_method 0 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:44.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a7531c700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c004810 0x7f5a7014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a7531c700 10 monclient(hunting): get_auth_request con 0x7f5a5c004810 auth_method 0 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a7531c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a7531c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a7531c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient(hunting): handle_auth_done global_id 4644 payload 824 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 -- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 msgr2=0x7f5a700a9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 -- 172.21.15.115:0/1384783921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c004810 msgr2=0x7f5a7014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c004810 0x7f5a7014e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient: _finish_hunting 0 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 monclient: found mon.noname-a 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 -- 172.21.15.115:0/1384783921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700a7b70 con 0x7f5a7013d6d0 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a75b1d700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a74b1b700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 0x7f5a700a90e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a60014000 tx=0x7f5a60015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 client.?.objecter ms_handle_connect 0x7f5a7013d6d0 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:44.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600010b0 con 0x7f5a7013d6d0 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: dump: 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:44.779 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: _finish_auth 0 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60001b30 con 0x7f5a7013d6d0 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 5 monclient: authenticate success, global_id 4644 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _renew_subs 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a701514c0 con 0x7f5a7013d6d0 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600070d0 con 0x7f5a7013d6d0 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:44.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a657fa700 10 monclient: dump: 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a677fe700 4 set_mon_vals no callback set 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:44.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _renew_subs 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a701519a0 con 0x7f5a7013d6d0 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.744+0000 7f5a77580700 1 librados: init done 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.745+0000 7f5a77580700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.745+0000 7f5a77580700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.745+0000 7f5a77580700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.745+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70072960 con 0x7f5a7013d6d0 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5a600074c0 con 0x7f5a7013d6d0 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 1 --2- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 0x7f5a5406ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:44.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5a60085220 con 0x7f5a7013d6d0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 10 client.4644.objecter ms_dispatch 0x7f5a70001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a7531c700 1 --2- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 0x7f5a5406ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 3 client.4644.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 3 client.4644.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a7531c700 10 monclient: get_auth_request con 0x7f5a540688b0 auth_method 0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.746+0000 7f5a657fa700 20 client.4644.objecter dump_active .. 0 homeless 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.747+0000 7f5a657fa700 10 client.4644.objecter ms_handle_connect 0x7f5a540688b0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.747+0000 7f5a7531c700 1 --2- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 0x7f5a5406ad60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006dc0 tx=0x7f5a6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.751+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a60052a40 con 0x7f5a7013d6d0 2023-09-30T01:05:44.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.751+0000 7f5a657fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:44.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.751+0000 7f5a657fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.971+0000 7f5a77580700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.971+0000 7f5a77580700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.971+0000 7f5a77580700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.971+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a70140260 con 0x7f5a7013d6d0 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.972+0000 7f5a657fa700 1 -- 172.21.15.115:0/1384783921 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5a60017100 con 0x7f5a7013d6d0 2023-09-30T01:05:44.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.972+0000 7f5a657fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:44.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.972+0000 7f5a657fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:44.973 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:44.973 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:44.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 10 librados: watch_flush enter 2023-09-30T01:05:44.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 10 librados: watch_flush exit 2023-09-30T01:05:44.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 20 client.4644.objecter shutdown clearing up homeless session... 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 10 client.4644.objecter successfully canceled tick 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 msgr2=0x7f5a5406ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 0x7f5a5406ad60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006dc0 tx=0x7f5a6c008040 comp rx=0 tx=0).stop 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 10 monclient: shutdown 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 msgr2=0x7f5a700a90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 0x7f5a700a90e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a60014000 tx=0x7f5a60015000 comp rx=0 tx=0).stop 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 shutdown_connections 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> v2:172.21.15.115:6816/115635 conn(0x7f5a540688b0 0x7f5a5406ad60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a7014ea80 0x7f5a700a9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c004810 0x7f5a7014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 --2- 172.21.15.115:0/1384783921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a7013d6d0 0x7f5a700a90e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 >> 172.21.15.115:0/1384783921 conn(0x7f5a7006a460 msgr2=0x7f5a700b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 shutdown_connections 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 -- 172.21.15.115:0/1384783921 wait complete. 2023-09-30T01:05:44.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:44.974+0000 7f5a77580700 1 librados: shutdown 2023-09-30T01:05:47.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:48.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.115+0000 7f6bf95c4700 10 monclient: get_monmap_and_config 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.115+0000 7f6bf95c4700 10 monclient: build_initial_monmap 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.115+0000 7f6bf95c4700 10 monclient: monmap: 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:48.116813+0000 2023-09-30T01:05:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:48.116813+0000 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:48.117 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.116+0000 7f6bf95c4700 1 Processor -- start 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.116+0000 7f6bf95c4700 1 -- start start 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.116+0000 7f6bf95c4700 10 monclient: init 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:05:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 0x7f6bf41415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-c con 0x7f6bf40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-b con 0x7f6bf40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 0x7f6bf40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-a con 0x7f6bf40a5c80 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf40a7040 con 0x7f6bf40a5c80 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf40a5480 con 0x7f6bf40ae8f0 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40a5760 con 0x7f6bf40aecc0 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient(hunting): authenticate will time out at 1209.567189s 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 0x7f6bf40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:33996/0 (socket says 172.21.15.115:33996) 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 -- 172.21.15.115:0/3297096851 learned_addr learned my addr 172.21.15.115:0/3297096851 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): get_auth_request con 0x7f6bf40aecc0 auth_method 0 2023-09-30T01:05:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): get_auth_request con 0x7f6bf40a5c80 auth_method 0 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf2ffd700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 0x7f6bf41415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_done global_id 4487 payload 274 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 -- 172.21.15.115:0/3297096851 >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 msgr2=0x7f6bf41415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 0x7f6bf41415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 -- 172.21.15.115:0/3297096851 >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 msgr2=0x7f6bf40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 0x7f6bf40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient: _finish_hunting 0 2023-09-30T01:05:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 monclient: found mon.noname-b 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 -- 172.21.15.115:0/3297096851 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4141b00 con 0x7f6bf40aecc0 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf2ffd700 10 monclient: get_auth_request con 0x7f6bf40ae8f0 auth_method 0 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf2ffd700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 0x7f6bf41415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf37fe700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 0x7f6bf40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/3297096851 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1d50 tx=0x7f6be4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e32187a4d01e1e2a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 1 -- 172.21.15.115:0/3297096851 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4009e70 con 0x7f6bf40aecc0 2023-09-30T01:05:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: dump: 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 5 monclient: authenticate success, global_id 4487 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 1 -- 172.21.15.115:0/3297096851 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be400b860 con 0x7f6bf40aecc0 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf95c4700 4 set_mon_vals no callback set 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 1 -- 172.21.15.115:0/3297096851 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be400bc00 con 0x7f6bf40aecc0 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.117+0000 7f6bf1ffb700 10 monclient: dump: 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 10 monclient: shutdown 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 -- 172.21.15.115:0/3297096851 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 msgr2=0x7f6bf40b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/3297096851 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1d50 tx=0x7f6be4009250 comp rx=0 tx=0).stop 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 -- 172.21.15.115:0/3297096851 shutdown_connections 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.115:3300/0 conn(0x7f6bf40a5c80 0x7f6bf40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/3297096851 >> v2:172.21.15.157:3300/0 conn(0x7f6bf40ae8f0 0x7f6bf41415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/3297096851 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40aecc0 0x7f6bf40b1430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 -- 172.21.15.115:0/3297096851 >> 172.21.15.115:0/3297096851 conn(0x7f6bf406a490 msgr2=0x7f6bf40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 -- 172.21.15.115:0/3297096851 shutdown_connections 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.118+0000 7f6bf95c4700 1 -- 172.21.15.115:0/3297096851 wait complete. 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 10 monclient: build_initial_monmap 2023-09-30T01:05:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 10 monclient: monmap: 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 librados: starting msgr at 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 librados: starting objecter 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 Processor -- start 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 -- start start 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 librados: setting wanted keys 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 1 librados: calling monclient init 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.119+0000 7f6bf95c4700 10 monclient: init 2023-09-30T01:05:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-a con 0x7f6bf40a5c80 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-c con 0x7f6bf40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bf40aecc0 0x7f6bf4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): picked mon.noname-b con 0x7f6bf40aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40aab50 con 0x7f6bf40a5c80 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40a8d50 con 0x7f6bf40aecc0 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40aa560 con 0x7f6bf40ae8f0 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient(hunting): authenticate will time out at 1209.570189s 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54796/0 (socket says 172.21.15.115:54796) 2023-09-30T01:05:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 -- 172.21.15.115:0/2853772129 learned_addr learned my addr 172.21.15.115:0/2853772129 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34004/0 (socket says 172.21.15.115:34004) 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): get_auth_request con 0x7f6bf40a5c80 auth_method 0 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): get_auth_request con 0x7f6bf40ae8f0 auth_method 0 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf37fe700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bf40aecc0 0x7f6bf4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient(hunting): handle_auth_done global_id 4650 payload 824 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 -- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 msgr2=0x7f6bf40aa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 -- 172.21.15.115:0/2853772129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bf40aecc0 msgr2=0x7f6bf4151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bf40aecc0 0x7f6bf4151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 monclient: found mon.noname-a 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 -- 172.21.15.115:0/2853772129 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec009a20 con 0x7f6bf40a5c80 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf2ffd700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c830 tx=0x7f6be8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 client.?.objecter ms_handle_connect 0x7f6bf40a5c80 2023-09-30T01:05:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011600 con 0x7f6bf40a5c80 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: dump: 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: _finish_auth 0 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8011760 con 0x7f6bf40a5c80 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 5 monclient: authenticate success, global_id 4650 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient: _renew_subs 2023-09-30T01:05:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be40094f0 con 0x7f6bf40a5c80 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011b20 con 0x7f6bf40a5c80 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bdaffd700 10 monclient: dump: 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.120+0000 7f6bf17fa700 4 set_mon_vals no callback set 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 client.4650.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 monclient: _renew_subs 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4151d90 con 0x7f6bf40a5c80 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 1 librados: init done 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.121+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4072960 con 0x7f6bf40a5c80 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6be80193b0 con 0x7f6bf40a5c80 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 1 --2- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 0x7f6be006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6be8083ea0 con 0x7f6bf40a5c80 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 10 client.4650.objecter ms_dispatch 0x7f6bf4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 3 client.4650.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 3 client.4650.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 0x7f6be006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bf27fc700 10 monclient: get_auth_request con 0x7f6be0068890 auth_method 0 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.122+0000 7f6bdaffd700 20 client.4650.objecter dump_active .. 0 homeless 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.123+0000 7f6bdaffd700 10 client.4650.objecter ms_handle_connect 0x7f6be0068890 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.123+0000 7f6bf27fc700 1 --2- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 0x7f6be006ad40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1d50 tx=0x7f6be4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.127+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6be8051760 con 0x7f6bf40a5c80 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.127+0000 7f6bdaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.127+0000 7f6bdaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.348+0000 7f6bf95c4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.348+0000 7f6bf95c4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.348+0000 7f6bf95c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.348+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bf40b1970 con 0x7f6bf40a5c80 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.349+0000 7f6bdaffd700 1 -- 172.21.15.115:0/2853772129 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6be8015100 con 0x7f6bf40a5c80 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.349+0000 7f6bdaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.349+0000 7f6bdaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:48.350 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:48.350 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.350+0000 7f6bf95c4700 10 librados: watch_flush enter 2023-09-30T01:05:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 10 librados: watch_flush exit 2023-09-30T01:05:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 20 client.4650.objecter shutdown clearing up homeless session... 2023-09-30T01:05:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 10 client.4650.objecter successfully canceled tick 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 msgr2=0x7f6be006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 0x7f6be006ad40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1d50 tx=0x7f6be4017000 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 10 monclient: shutdown 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 msgr2=0x7f6bf4140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c830 tx=0x7f6be8008040 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 shutdown_connections 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> v2:172.21.15.115:6816/115635 conn(0x7f6be0068890 0x7f6be006ad40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bf40aecc0 0x7f6bf4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bf40a5c80 0x7f6bf4140090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 --2- 172.21.15.115:0/2853772129 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bf40ae8f0 0x7f6bf40aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 >> 172.21.15.115:0/2853772129 conn(0x7f6bf406a490 msgr2=0x7f6bf40adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:48.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 shutdown_connections 2023-09-30T01:05:48.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 -- 172.21.15.115:0/2853772129 wait complete. 2023-09-30T01:05:48.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:48.351+0000 7f6bf95c4700 1 librados: shutdown 2023-09-30T01:05:51.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.491+0000 7f4ed8ecc700 10 monclient: get_monmap_and_config 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.491+0000 7f4ed8ecc700 10 monclient: build_initial_monmap 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.491+0000 7f4ed8ecc700 10 monclient: monmap: 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:51.492800+0000 2023-09-30T01:05:51.492 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:51.492800+0000 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.492+0000 7f4ed8ecc700 1 Processor -- start 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.492+0000 7f4ed8ecc700 1 -- start start 2023-09-30T01:05:51.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.492+0000 7f4ed8ecc700 10 monclient: init 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed40ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ed40aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed41415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40a7040 con 0x7f4ed41415c0 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40a57f0 con 0x7f4ed40aecc0 2023-09-30T01:05:51.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed40a5480 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed8ecc700 10 monclient(hunting): authenticate will time out at 1212.943264s 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed2d9d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f4ed41415c0 0x7f4ed40adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57040/0 (socket says 172.21.15.115:57040) 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59202/0 (socket says 172.21.15.115:59202) 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed2d9d700 1 -- 172.21.15.115:0/2699362906 learned_addr learned my addr 172.21.15.115:0/2699362906 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): get_auth_request con 0x7f4ed40ae8f0 auth_method 0 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:51.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient(hunting): handle_auth_done global_id 4653 payload 274 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 -- 172.21.15.115:0/2699362906 >> v1:172.21.15.157:6789/0 conn(0x7f4ed40aecc0 legacy=0x7f4ed40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 -- 172.21.15.115:0/2699362906 >> v1:172.21.15.115:6789/0 conn(0x7f4ed41415c0 legacy=0x7f4ed40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient: _finish_hunting 0 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 monclient: found mon.noname-a 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed159a700 1 -- 172.21.15.115:0/2699362906 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152376328 0 0) 0x7f4ed40a7040 con 0x7f4ed41415c0 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 -- 172.21.15.115:0/2699362906 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40a55c0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.493+0000 7f4ed259c700 1 --2- 172.21.15.115:0/2699362906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0098e0 tx=0x7f4ebc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377e28879935c9e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 1 -- 172.21.15.115:0/2699362906 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc009e80 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: dump: 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.505 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: _finish_auth 0 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 5 monclient: authenticate success, global_id 4653 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 1 -- 172.21.15.115:0/2699362906 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc00b860 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 4 set_mon_vals no callback set 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 1 -- 172.21.15.115:0/2699362906 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00bbe0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed159a700 10 monclient: dump: 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:51.507 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 10 monclient: shutdown 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/2699362906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40ae8f0 msgr2=0x7f4ed40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/2699362906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0098e0 tx=0x7f4ebc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/2699362906 shutdown_connections 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/2699362906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40ae8f0 0x7f4ed40a5d10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/2699362906 >> 172.21.15.115:0/2699362906 conn(0x7f4ed406a460 msgr2=0x7f4ed40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/2699362906 shutdown_connections 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.494+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/2699362906 wait complete. 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 10 monclient: build_initial_monmap 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 10 monclient: monmap: 2023-09-30T01:05:51.508 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 librados: starting msgr at 2023-09-30T01:05:51.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 librados: starting objecter 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 Processor -- start 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 -- start start 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 librados: setting wanted keys 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 1 librados: calling monclient init 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.495+0000 7f4ed8ecc700 10 monclient: init 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-c con 0x7f4ed40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40aecc0 0x7f4ed413e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed40aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:51.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 0x7f4ed4151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): picked mon.noname-b con 0x7f4ed41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40a9e20 con 0x7f4ed40aecc0 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40ab4b0 con 0x7f4ed41415c0 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40ab720 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient(hunting): authenticate will time out at 1212.946261s 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40aecc0 0x7f4ed413e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:51.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38504/0 (socket says 172.21.15.115:38504) 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 -- 172.21.15.115:0/4242077691 learned_addr learned my addr 172.21.15.115:0/4242077691 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): get_auth_request con 0x7f4ed40ae8f0 auth_method 0 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): get_auth_request con 0x7f4ed40aecc0 auth_method 0 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:51.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 0x7f4ed4151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 10 monclient(hunting): get_auth_request con 0x7f4ed41415c0 auth_method 0 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 -- 172.21.15.115:0/4242077691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 msgr2=0x7f4ed4151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 0x7f4ed4151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 -- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40aecc0 msgr2=0x7f4ed413e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40aecc0 0x7f4ed413e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient: _finish_hunting 0 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 monclient: found mon.noname-c 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 -- 172.21.15.115:0/4242077691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc009420 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed2d9d700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 0x7f4ed4151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed259c700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00b3c0 tx=0x7f4ebc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 client.?.objecter ms_handle_connect 0x7f4ed40ae8f0 2023-09-30T01:05:51.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc01f070 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 monclient: dump: 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:51.515 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 10 monclient: _finish_auth 0 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc000c90 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 5 monclient: authenticate success, global_id 4493 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.496+0000 7f4ed8ecc700 10 monclient: _renew_subs 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed413ee40 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4eca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00bbe0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4eca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:51.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4eca7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4eca7fc700 10 monclient: dump: 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:51.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed0d99700 4 set_mon_vals no callback set 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 monclient: _renew_subs 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40a9740 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 1 librados: init done 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.497+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed4072960 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4ebc0118a0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 1 --2- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 0x7f4eb406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:51.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4ebc0852d0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 10 client.4493.objecter ms_dispatch 0x7f4ed4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 3 client.4493.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 3 client.4493.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4ed1d9b700 1 --2- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 0x7f4eb406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4ed1d9b700 10 monclient: get_auth_request con 0x7f4eb4068890 auth_method 0 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.498+0000 7f4eca7fc700 20 client.4493.objecter dump_active .. 0 homeless 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.499+0000 7f4eca7fc700 10 client.4493.objecter ms_handle_connect 0x7f4eb4068890 2023-09-30T01:05:51.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.499+0000 7f4ed1d9b700 1 --2- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 0x7f4eb406ad40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40ab090 tx=0x7f4ec4007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:51.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.503+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ebc052b90 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.503+0000 7f4eca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:51.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.503+0000 7f4eca7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:51.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4ed8ecc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:51.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4ed8ecc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4ed8ecc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ed40aa100 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4eca7fc700 1 -- 172.21.15.115:0/4242077691 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4ebc0561b0 con 0x7f4ed40ae8f0 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4eca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.724+0000 7f4eca7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:51.725 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 10 librados: watch_flush enter 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 10 librados: watch_flush exit 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 20 client.4493.objecter shutdown clearing up homeless session... 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 10 client.4493.objecter successfully canceled tick 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 msgr2=0x7f4eb406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 0x7f4eb406ad40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40ab090 tx=0x7f4ec4007ec0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 10 monclient: shutdown 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 msgr2=0x7f4ed40aa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00b3c0 tx=0x7f4ebc017000 comp rx=0 tx=0).stop 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 shutdown_connections 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> v2:172.21.15.115:6816/115635 conn(0x7f4eb4068890 0x7f4eb406ad40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed41415c0 0x7f4ed4151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f4ed40ae8f0 0x7f4ed40aa2b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 --2- 172.21.15.115:0/4242077691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ed40aecc0 0x7f4ed413e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 >> 172.21.15.115:0/4242077691 conn(0x7f4ed406a460 msgr2=0x7f4ed4144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 shutdown_connections 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 -- 172.21.15.115:0/4242077691 wait complete. 2023-09-30T01:05:51.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:51.726+0000 7f4ed8ecc700 1 librados: shutdown 2023-09-30T01:05:54.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 10 monclient: get_monmap_and_config 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 10 monclient: build_initial_monmap 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 10 monclient: monmap: 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:54.867 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:54.868269+0000 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:54.868269+0000 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:54.868 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 1 Processor -- start 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 1 -- start start 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.867+0000 7f341ca7b700 10 monclient: init 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34180ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34180aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f34180a5d10 0x7f34180adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-a con 0x7f34180a5d10 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180a6190 con 0x7f34180ae8f0 2023-09-30T01:05:54.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180a7040 con 0x7f34180aecc0 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3418141cd0 con 0x7f34180a5d10 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341ca7b700 10 monclient(hunting): authenticate will time out at 1216.318706s 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f34180a5d10 0x7f34180adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341659c700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f34180ae8f0 0x7f34180b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57042/0 (socket says 172.21.15.115:57042) 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f341659c700 1 -- 172.21.15.115:0/2748914972 learned_addr learned my addr 172.21.15.115:0/2748914972 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): get_auth_request con 0x7f34180a5d10 auth_method 0 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:54.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.868+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 10 monclient(hunting): handle_auth_done global_id 4662 payload 274 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 1 -- 172.21.15.115:0/2748914972 >> v1:172.21.15.157:6789/0 conn(0x7f34180aecc0 legacy=0x7f34181415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 1 -- 172.21.15.115:0/2748914972 >> v1:172.21.15.115:6789/0 conn(0x7f34180ae8f0 legacy=0x7f34180b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 1 monclient: found mon.noname-a 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 1 -- 172.21.15.115:0/2748914972 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653446996 0 0) 0x7f340c001110 con 0x7f34180ae8f0 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 1 -- 172.21.15.115:0/2748914972 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180a57f0 con 0x7f34180a5d10 2023-09-30T01:05:54.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f3416d9d700 1 --2- 172.21.15.115:0/2748914972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f34180adb60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e60 tx=0x7f3410009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9642a94f0beac382 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 1 -- 172.21.15.115:0/2748914972 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341000e070 con 0x7f34180a5d10 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: dump: 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.880 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: _finish_auth 0 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341ca7b700 5 monclient: authenticate success, global_id 4662 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 1 -- 172.21.15.115:0/2748914972 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341000bbf0 con 0x7f34180a5d10 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341ca7b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341ca7b700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:54.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341ca7b700 4 set_mon_vals no callback set 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 1 -- 172.21.15.115:0/2748914972 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410012430 con 0x7f34180a5d10 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341559a700 10 monclient: dump: 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:54.882 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.869+0000 7f341ca7b700 10 monclient: shutdown 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 -- 172.21.15.115:0/2748914972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 msgr2=0x7f34180adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 --2- 172.21.15.115:0/2748914972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f34180adb60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e60 tx=0x7f3410009780 comp rx=0 tx=0).stop 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 -- 172.21.15.115:0/2748914972 shutdown_connections 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 --2- 172.21.15.115:0/2748914972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f34180adb60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 -- 172.21.15.115:0/2748914972 >> 172.21.15.115:0/2748914972 conn(0x7f341806a460 msgr2=0x7f34180630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 -- 172.21.15.115:0/2748914972 shutdown_connections 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 1 -- 172.21.15.115:0/2748914972 wait complete. 2023-09-30T01:05:54.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 10 monclient: build_initial_monmap 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.870+0000 7f341ca7b700 10 monclient: monmap: 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 librados: starting msgr at 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 librados: starting objecter 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 Processor -- start 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 -- start start 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 librados: setting wanted keys 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 1 librados: calling monclient init 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.871+0000 7f341ca7b700 10 monclient: init 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-a con 0x7f34180a5d10 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 0x7f34180a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:54.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-b con 0x7f34180ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34180aecc0 0x7f34180a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): picked mon.noname-c con 0x7f34180aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180a9a20 con 0x7f34180a5d10 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180a9720 con 0x7f34180ae8f0 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180ab8b0 con 0x7f34180aecc0 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341ca7b700 10 monclient(hunting): authenticate will time out at 1216.322352s 2023-09-30T01:05:54.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59224/0 (socket says 172.21.15.115:59224) 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34180aecc0 0x7f34180a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 -- 172.21.15.115:0/1799580079 learned_addr learned my addr 172.21.15.115:0/1799580079 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): get_auth_request con 0x7f34180a5d10 auth_method 0 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): get_auth_request con 0x7f34180aecc0 auth_method 0 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:54.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3416d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3415d9b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 0x7f34180a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 -- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34180aecc0 msgr2=0x7f34180a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34180aecc0 0x7f34180a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 -- 172.21.15.115:0/1799580079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 msgr2=0x7f34180a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 0x7f34180a91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:54.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient: _finish_hunting 0 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 monclient: found mon.noname-a 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 -- 172.21.15.115:0/1799580079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3410009a20 con 0x7f34180a5d10 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3415d9b700 10 monclient: get_auth_request con 0x7f34180ae8f0 auth_method 0 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f3415d9b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 0x7f34180a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f341659c700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f340c00c840 tx=0x7f340c007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 10 client.?.objecter ms_handle_connect 0x7f34180a5d10 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c005b30 con 0x7f34180a5d10 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:54.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.872+0000 7f34067fc700 10 monclient: dump: 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 10 monclient: _finish_auth 0 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c005c90 con 0x7f34180a5d10 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 5 monclient: authenticate success, global_id 4668 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 10 monclient: _renew_subs 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341813ef20 con 0x7f34180a5d10 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c007300 con 0x7f34180a5d10 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f34067fc700 10 monclient: dump: 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f3414d99700 4 set_mon_vals no callback set 2023-09-30T01:05:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 10 monclient: _renew_subs 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34180aa410 con 0x7f34180a5d10 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.873+0000 7f341ca7b700 1 librados: init done 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f341ca7b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f341ca7b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f341ca7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34180a9d20 con 0x7f34180a5d10 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f340c0076f0 con 0x7f34180a5d10 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 1 --2- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 0x7f33f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f3415d9b700 1 --2- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 0x7f33f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f340c083820 con 0x7f34180a5d10 2023-09-30T01:05:54.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 10 client.4668.objecter ms_dispatch 0x7f3418001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 3 client.4668.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f34067fc700 3 client.4668.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.874+0000 7f3415d9b700 10 monclient: get_auth_request con 0x7f33f8068890 auth_method 0 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.875+0000 7f34067fc700 20 client.4668.objecter dump_active .. 0 homeless 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.875+0000 7f34067fc700 10 client.4668.objecter ms_handle_connect 0x7f33f8068890 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.875+0000 7f3415d9b700 1 --2- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 0x7f33f806ad40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.879+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f340c050f90 con 0x7f34180a5d10 2023-09-30T01:05:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.879+0000 7f34067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:54.879+0000 7f34067fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:55.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f341ca7b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f341ca7b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f341ca7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f34180a6190 con 0x7f34180a5d10 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f34067fc700 1 -- 172.21.15.115:0/1799580079 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f34180a6190 con 0x7f34180a5d10 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f34067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.098+0000 7f34067fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:55.100 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:55.100 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:55.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 10 librados: watch_flush enter 2023-09-30T01:05:55.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 10 librados: watch_flush exit 2023-09-30T01:05:55.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 20 client.4668.objecter shutdown clearing up homeless session... 2023-09-30T01:05:55.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 10 client.4668.objecter successfully canceled tick 2023-09-30T01:05:55.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 msgr2=0x7f33f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 0x7f33f806ad40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).stop 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 10 monclient: shutdown 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 msgr2=0x7f341813e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f340c00c840 tx=0x7f340c007c50 comp rx=0 tx=0).stop 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 shutdown_connections 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> v2:172.21.15.115:6816/115635 conn(0x7f33f8068890 0x7f33f806ad40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34180aecc0 0x7f34180a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34180a5d10 0x7f341813e390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 --2- 172.21.15.115:0/1799580079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34180ae8f0 0x7f34180a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:55.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.100+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 >> 172.21.15.115:0/1799580079 conn(0x7f341806a460 msgr2=0x7f3418144890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:55.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.101+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 shutdown_connections 2023-09-30T01:05:55.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.101+0000 7f341ca7b700 1 -- 172.21.15.115:0/1799580079 wait complete. 2023-09-30T01:05:55.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:55.101+0000 7f341ca7b700 1 librados: shutdown 2023-09-30T01:05:58.113 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 10 monclient: get_monmap_and_config 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 10 monclient: build_initial_monmap 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 10 monclient: monmap: 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:05:58.242093+0000 2023-09-30T01:05:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:05:58.242093+0000 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:05:58.242 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 1 Processor -- start 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 1 -- start start 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.241+0000 7fe643d5c700 10 monclient: init 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-b con 0x7fe63c0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe63c0aecc0 0x7fe63c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-a con 0x7fe63c0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe63c1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:05:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c0a7040 con 0x7fe63c1415c0 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63c0b19b0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c0a57f0 con 0x7fe63c0ae8f0 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe643d5c700 10 monclient(hunting): authenticate will time out at 1219.692628s 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe63c0aecc0 0x7fe63c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe63c0aecc0 0x7fe63c0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59240/0 (socket says 172.21.15.115:59240) 2023-09-30T01:05:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38520/0 (socket says 172.21.15.115:38520) 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 1 -- 172.21.15.115:0/2121241855 learned_addr learned my addr 172.21.15.115:0/2121241855 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): get_auth_request con 0x7fe63c0aecc0 auth_method 0 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): get_auth_request con 0x7fe63c0ae8f0 auth_method 0 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:58.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.242+0000 7fe6412f7700 10 monclient(hunting): handle_auth_done global_id 4671 payload 274 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 -- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 msgr2=0x7fe63c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 -- 172.21.15.115:0/2121241855 >> v1:172.21.15.157:6789/0 conn(0x7fe63c1415c0 legacy=0x7fe63c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 10 monclient: _finish_hunting 0 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 monclient: found mon.noname-a 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:05:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 -- 172.21.15.115:0/2121241855 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0a5480 con 0x7fe63c0aecc0 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe641af8700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe6412f7700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c0aecc0 0x7fe63c0b1470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe6300099b0 tx=0x7fe630009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849a06dc260ae327 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 1 -- 172.21.15.115:0/2121241855 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630009ea0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: dump: 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: _finish_auth 0 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 5 monclient: authenticate success, global_id 4671 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 1 -- 172.21.15.115:0/2121241855 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63000b820 con 0x7fe63c0aecc0 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 10 monclient: get_monmap_and_config success 2023-09-30T01:05:58.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 4 set_mon_vals no callback set 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 1 -- 172.21.15.115:0/2121241855 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000bbc0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe640af6700 10 monclient: dump: 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:58.258 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 10 monclient: shutdown 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 -- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c0aecc0 msgr2=0x7fe63c0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c0aecc0 0x7fe63c0b1470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe6300099b0 tx=0x7fe630009250 comp rx=0 tx=0).stop 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 -- 172.21.15.115:0/2121241855 shutdown_connections 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0ae8f0 0x7fe63c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 --2- 172.21.15.115:0/2121241855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c0aecc0 0x7fe63c0b1470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.243+0000 7fe643d5c700 1 -- 172.21.15.115:0/2121241855 >> 172.21.15.115:0/2121241855 conn(0x7fe63c06a460 msgr2=0x7fe63c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 1 -- 172.21.15.115:0/2121241855 shutdown_connections 2023-09-30T01:05:58.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 1 -- 172.21.15.115:0/2121241855 wait complete. 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 10 monclient: build_initial_monmap 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 10 monclient: monmap: 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:05:58.260 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 1 librados: starting msgr at 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 1 librados: starting objecter 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.244+0000 7fe643d5c700 1 Processor -- start 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 -- start start 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 librados: setting wanted keys 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 librados: calling monclient init 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient: init 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient: _reopen_session rank -1 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe63c0ae8f0 0x7fe63c0a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-b con 0x7fe63c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:05:58.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-c con 0x7fe63c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 0x7fe63c151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): picked mon.noname-a con 0x7fe63c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c0a9a10 con 0x7fe63c1415c0 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c1513e0 con 0x7fe63c0ae8f0 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c0a8d10 con 0x7fe63c0aecc0 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): _renew_subs 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe643d5c700 10 monclient(hunting): authenticate will time out at 1219.695629s 2023-09-30T01:05:58.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38530/0 (socket says 172.21.15.115:38530) 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 1 -- 172.21.15.115:0/1873658858 learned_addr learned my addr 172.21.15.115:0/1873658858 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): get_auth_request con 0x7fe63c0aecc0 auth_method 0 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 0x7fe63c151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): get_auth_request con 0x7fe63c1415c0 auth_method 0 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:58.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe63c0ae8f0 0x7fe63c0a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6422f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): get_auth_request con 0x7fe63c0ae8f0 auth_method 0 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:05:58.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe641af8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.245+0000 7fe6412f7700 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 -- 172.21.15.115:0/1873658858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe63c0ae8f0 msgr2=0x7fe63c0a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe63c0ae8f0 0x7fe63c0a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 -- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 msgr2=0x7fe63c151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 0x7fe63c151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 10 monclient: _finish_hunting 0 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 monclient: found mon.noname-c 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 -- 172.21.15.115:0/1873658858 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe638009420 con 0x7fe63c0aecc0 2023-09-30T01:05:58.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6422f9700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 0x7fe63c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe6412f7700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe630009290 tx=0x7fe630016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 client.?.objecter ms_handle_connect 0x7fe63c0aecc0 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630011da0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: dump: 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.266 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: _finish_auth 0 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 5 monclient: authenticate success, global_id 4502 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63001b470 con 0x7fe63c0aecc0 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63001b830 con 0x7fe63c0aecc0 2023-09-30T01:05:58.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 10 monclient: _renew_subs 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300094f0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62ffff700 4 set_mon_vals no callback set 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe62dffb700 10 monclient: dump: 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 10 monclient: _renew_subs 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c13dee0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.246+0000 7fe643d5c700 1 librados: init done 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.247+0000 7fe643d5c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:58.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.247+0000 7fe643d5c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.247+0000 7fe643d5c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.247+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63c13e0d0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.247+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe63001bc20 con 0x7fe63c0aecc0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 1 --2- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 0x7fe62006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe630085b30 con 0x7fe63c0aecc0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 10 client.4502.objecter ms_dispatch 0x7fe63c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 3 client.4502.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 3 client.4502.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe641af8700 1 --2- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 0x7fe62006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe641af8700 10 monclient: get_auth_request con 0x7fe620068890 auth_method 0 2023-09-30T01:05:58.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 20 client.4502.objecter dump_active .. 0 homeless 2023-09-30T01:05:58.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe62dffb700 10 client.4502.objecter ms_handle_connect 0x7fe620068890 2023-09-30T01:05:58.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.248+0000 7fe641af8700 1 --2- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 0x7fe62006ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe638009f40 tx=0x7fe6380091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:05:58.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.252+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe63c13e0d0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.252+0000 7fe62dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:05:58.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.252+0000 7fe62dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:05:58.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.476+0000 7fe643d5c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.476+0000 7fe643d5c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.476+0000 7fe643d5c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.476+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe63c13e0d0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.477+0000 7fe62dffb700 1 -- 172.21.15.115:0/1873658858 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe63c13e0d0 con 0x7fe63c0aecc0 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.477+0000 7fe62dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.477+0000 7fe62dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:05:58.477 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:05:58.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.478+0000 7fe643d5c700 10 librados: watch_flush enter 2023-09-30T01:05:58.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.478+0000 7fe643d5c700 10 librados: watch_flush exit 2023-09-30T01:05:58.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.478+0000 7fe643d5c700 20 client.4502.objecter shutdown clearing up homeless session... 2023-09-30T01:05:58.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.478+0000 7fe643d5c700 10 client.4502.objecter successfully canceled tick 2023-09-30T01:05:58.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 msgr2=0x7fe62006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 0x7fe62006ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe638009f40 tx=0x7fe6380091a0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 10 monclient: shutdown 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 msgr2=0x7fe63c0a87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe630009290 tx=0x7fe630016000 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 shutdown_connections 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> v2:172.21.15.115:6816/115635 conn(0x7fe620068890 0x7fe62006ad40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe63c1415c0 0x7fe63c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe63c0ae8f0 0x7fe63c0a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 --2- 172.21.15.115:0/1873658858 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe63c0aecc0 0x7fe63c0a87d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 >> 172.21.15.115:0/1873658858 conn(0x7fe63c06a460 msgr2=0x7fe63c13ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:05:58.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 shutdown_connections 2023-09-30T01:05:58.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 -- 172.21.15.115:0/1873658858 wait complete. 2023-09-30T01:05:58.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:05:58.479+0000 7fe643d5c700 1 librados: shutdown 2023-09-30T01:06:01.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:01.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.619+0000 7f5a3a0a3700 10 monclient: get_monmap_and_config 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.619+0000 7f5a3a0a3700 10 monclient: build_initial_monmap 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.619+0000 7f5a3a0a3700 10 monclient: monmap: 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:01.620668+0000 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:01.620668+0000 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:01.621 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:01.621 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.620+0000 7f5a3a0a3700 1 Processor -- start 2023-09-30T01:06:01.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.620+0000 7f5a3a0a3700 1 -- start start 2023-09-30T01:06:01.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.620+0000 7f5a3a0a3700 10 monclient: init 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5a340ae8f0 0x7f5a340a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-c con 0x7f5a340ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5a340aecc0 0x7f5a340b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-a con 0x7f5a340aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a341415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340a7040 con 0x7f5a341415c0 2023-09-30T01:06:01.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a340b19b0 con 0x7f5a340aecc0 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a340a57f0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient(hunting): authenticate will time out at 1223.071212s 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5a340aecc0 0x7f5a340b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5a340aecc0 0x7f5a340b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:38688/0 (socket says 172.21.15.115:38688) 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 -- 172.21.15.115:0/668176691 learned_addr learned my addr 172.21.15.115:0/668176691 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): get_auth_request con 0x7f5a340aecc0 auth_method 0 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:01.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a337fe700 1 --2- 172.21.15.115:0/668176691 >> v2:172.21.15.157:3300/0 conn(0x7f5a340ae8f0 0x7f5a340a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a337fe700 10 monclient(hunting): get_auth_request con 0x7f5a340ae8f0 auth_method 0 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_done global_id 4677 payload 274 2023-09-30T01:06:01.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 -- 172.21.15.115:0/668176691 >> v2:172.21.15.157:3300/0 conn(0x7f5a340ae8f0 msgr2=0x7f5a340a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 --2- 172.21.15.115:0/668176691 >> v2:172.21.15.157:3300/0 conn(0x7f5a340ae8f0 0x7f5a340a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 -- 172.21.15.115:0/668176691 >> v1:172.21.15.157:6789/0 conn(0x7f5a341415c0 legacy=0x7f5a340adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 monclient: found mon.noname-a 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 -- 172.21.15.115:0/668176691 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340a5480 con 0x7f5a340aecc0 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a32ffd700 1 --2- 172.21.15.115:0/668176691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340aecc0 0x7f5a340b1470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b59c0 tx=0x7f5a24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332f30aef18e5763 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 1 -- 172.21.15.115:0/668176691 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2400b860 con 0x7f5a340aecc0 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: dump: 2023-09-30T01:06:01.633 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: _finish_auth 0 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:01.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 1 -- 172.21.15.115:0/668176691 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2400b9c0 con 0x7f5a340aecc0 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 5 monclient: authenticate success, global_id 4677 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 1 -- 172.21.15.115:0/668176691 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2400bd60 con 0x7f5a340aecc0 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a3a0a3700 4 set_mon_vals no callback set 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.621+0000 7f5a327fc700 10 monclient: dump: 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:01.635 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 10 monclient: shutdown 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/668176691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340aecc0 msgr2=0x7f5a340b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/668176691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340aecc0 0x7f5a340b1470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b59c0 tx=0x7f5a24009250 comp rx=0 tx=0).stop 2023-09-30T01:06:01.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/668176691 shutdown_connections 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/668176691 >> v2:172.21.15.157:3300/0 conn(0x7f5a340ae8f0 0x7f5a340a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/668176691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340aecc0 0x7f5a340b1470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/668176691 >> 172.21.15.115:0/668176691 conn(0x7f5a3406a460 msgr2=0x7f5a340630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/668176691 shutdown_connections 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.622+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/668176691 wait complete. 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 10 monclient: build_initial_monmap 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 10 monclient: monmap: 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:01.637 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 librados: starting msgr at 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 librados: starting objecter 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 Processor -- start 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 -- start start 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 librados: setting wanted keys 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 1 librados: calling monclient init 2023-09-30T01:06:01.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.623+0000 7f5a3a0a3700 10 monclient: init 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-a con 0x7f5a340ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a340aecc0 0x7f5a34151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-b con 0x7f5a340aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 0x7f5a341518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): picked mon.noname-c con 0x7f5a341415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a340abc20 con 0x7f5a340ae8f0 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a340aa340 con 0x7f5a340aecc0 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a340a88c0 con 0x7f5a341415c0 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 10 monclient(hunting): authenticate will time out at 1223.074211s 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 0x7f5a341518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:38698/0 (socket says 172.21.15.115:38698) 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 -- 172.21.15.115:0/3954162816 learned_addr learned my addr 172.21.15.115:0/3954162816 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): get_auth_request con 0x7f5a340ae8f0 auth_method 0 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:01.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): get_auth_request con 0x7f5a341415c0 auth_method 0 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a340aecc0 0x7f5a34151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): get_auth_request con 0x7f5a340aecc0 auth_method 0 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:01.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 -- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 msgr2=0x7f5a341518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 0x7f5a341518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 -- 172.21.15.115:0/3954162816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a340aecc0 msgr2=0x7f5a34151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a340aecc0 0x7f5a34151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient: _finish_hunting 0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 monclient: found mon.noname-a 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 -- 172.21.15.115:0/3954162816 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240094f0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a33fff700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 0x7f5a341518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a337fe700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800d890 tx=0x7f5a28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 client.?.objecter ms_handle_connect 0x7f5a340ae8f0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a280125f0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:01.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 monclient: dump: 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:06:01.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28012750 con 0x7f5a340ae8f0 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.624+0000 7f5a3a0a3700 5 monclient: authenticate success, global_id 4680 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _renew_subs 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a340a94e0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a1b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28012af0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a1b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a1b7fe700 10 monclient: dump: 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a31ffb700 4 set_mon_vals no callback set 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _renew_subs 2023-09-30T01:06:01.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a340aaa40 con 0x7f5a340ae8f0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 1 librados: init done 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.625+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34072960 con 0x7f5a340ae8f0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5a2801e460 con 0x7f5a340ae8f0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 1 --2- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 0x7f5a2006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a32ffd700 1 --2- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 0x7f5a2006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5a28084400 con 0x7f5a340ae8f0 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 10 client.4680.objecter ms_dispatch 0x7f5a34001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:01.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 3 client.4680.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a1b7fe700 3 client.4680.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.626+0000 7f5a32ffd700 10 monclient: get_auth_request con 0x7f5a200688c0 auth_method 0 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.627+0000 7f5a1b7fe700 20 client.4680.objecter dump_active .. 0 homeless 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.627+0000 7f5a1b7fe700 10 client.4680.objecter ms_handle_connect 0x7f5a200688c0 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.627+0000 7f5a32ffd700 1 --2- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 0x7f5a2006ad70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b59c0 tx=0x7f5a24017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:01.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.631+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a28051cc0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.631+0000 7f5a1b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:01.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.631+0000 7f5a1b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:01.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a3a0a3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:01.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a3a0a3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:01.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a3a0a3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:01.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a340aa620 con 0x7f5a340ae8f0 2023-09-30T01:06:01.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a1b7fe700 1 -- 172.21.15.115:0/3954162816 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5a280552e0 con 0x7f5a340ae8f0 2023-09-30T01:06:01.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a1b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:01.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.850+0000 7f5a1b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:01.853 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:01.853 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 10 librados: watch_flush enter 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 10 librados: watch_flush exit 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 20 client.4680.objecter shutdown clearing up homeless session... 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 10 client.4680.objecter successfully canceled tick 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 msgr2=0x7f5a2006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 0x7f5a2006ad70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b59c0 tx=0x7f5a24017000 comp rx=0 tx=0).stop 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 10 monclient: shutdown 2023-09-30T01:06:01.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 msgr2=0x7f5a3413e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800d890 tx=0x7f5a28008040 comp rx=0 tx=0).stop 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 shutdown_connections 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> v2:172.21.15.115:6816/115635 conn(0x7f5a200688c0 0x7f5a2006ad70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5a341415c0 0x7f5a341518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5a340ae8f0 0x7f5a3413e630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 --2- 172.21.15.115:0/3954162816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a340aecc0 0x7f5a34151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.852+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 >> 172.21.15.115:0/3954162816 conn(0x7f5a3406a460 msgr2=0x7f5a34144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:01.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.853+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 shutdown_connections 2023-09-30T01:06:01.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.853+0000 7f5a3a0a3700 1 -- 172.21.15.115:0/3954162816 wait complete. 2023-09-30T01:06:01.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:01.853+0000 7f5a3a0a3700 1 librados: shutdown 2023-09-30T01:06:04.865 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:04.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.994+0000 7f2693d56700 10 monclient: get_monmap_and_config 2023-09-30T01:06:04.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.994+0000 7f2693d56700 10 monclient: build_initial_monmap 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.994+0000 7f2693d56700 10 monclient: monmap: 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:04.995409+0000 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:04.995409+0000 2023-09-30T01:06:04.995 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:04.996 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:04.996 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:04.996 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:04.996 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:04.996 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:04.997 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:04.997 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 Processor -- start 2023-09-30T01:06:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 -- start start 2023-09-30T01:06:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient: init 2023-09-30T01:06:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T01:06:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f268c0ae8f0 0x7f268c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-c con 0x7f268c0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f268c0aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f268c1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c0a7040 con 0x7f268c0aecc0 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c0a57f0 con 0x7f268c1415c0 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c0a5480 con 0x7f268c0ae8f0 2023-09-30T01:06:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.995+0000 7f2693d56700 10 monclient(hunting): authenticate will time out at 1226.445934s 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f268c0aecc0 0x7f268c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56672/0 (socket says 172.21.15.115:56672) 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 1 -- 172.21.15.115:0/2529361835 learned_addr learned my addr 172.21.15.115:0/2529361835 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 1 --2- 172.21.15.115:0/2529361835 >> v2:172.21.15.157:3300/0 conn(0x7f268c0ae8f0 0x7f268c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): get_auth_request con 0x7f268c0ae8f0 auth_method 0 2023-09-30T01:06:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762802541 0 0) 0x7f268c0a7040 con 0x7f268c0aecc0 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): my global_id is 4683 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2678003570 con 0x7f268c0aecc0 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922437140 0 0) 0x7f268c0a57f0 con 0x7f268c1415c0 2023-09-30T01:06:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): my global_id is 4597 2023-09-30T01:06:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f268c0a7040 con 0x7f268c1415c0 2023-09-30T01:06:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982335516 0 0) 0x7f2678003570 con 0x7f268c0aecc0 2023-09-30T01:06:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 >> v2:172.21.15.157:3300/0 conn(0x7f268c0ae8f0 msgr2=0x7f268c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 --2- 172.21.15.115:0/2529361835 >> v2:172.21.15.157:3300/0 conn(0x7f268c0ae8f0 0x7f268c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 >> v1:172.21.15.157:6789/0 conn(0x7f268c1415c0 legacy=0x7f268c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: _finish_hunting 0 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:06:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0a55c0 con 0x7f268c0aecc0 2023-09-30T01:06:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: _finish_auth 0 2023-09-30T01:06:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f2688001470 con 0x7f268c0aecc0 2023-09-30T01:06:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2693d56700 5 monclient: authenticate success, global_id 4683 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2693d56700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: dump: 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 >> v1:172.21.15.115:6789/0 conn(0x7f268c0aecc0 legacy=0x7f268c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:06:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678003bf0 0x7f2678003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): picked mon.c con 0x7f2678003bf0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 0x7f2678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): picked mon.a con 0x7f2678004810 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26780051c0 0x7f267800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): picked mon.b con 0x7f26780051c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0a5480 con 0x7f2678004810 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267800c0b0 con 0x7f26780051c0 2023-09-30T01:06:05.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267800c270 con 0x7f2678003bf0 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2690af0700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2693d56700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 0x7f2678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678003bf0 0x7f2678003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): get_auth_request con 0x7f2678004810 auth_method 0 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): get_auth_request con 0x7f2678003bf0 auth_method 0 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26780051c0 0x7f267800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26912f1700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): get_auth_request con 0x7f26780051c0 auth_method 0 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.996+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 10 monclient(hunting): handle_auth_done global_id 4683 payload 293 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 -- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678003bf0 msgr2=0x7f2678003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678003bf0 0x7f2678003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 -- 172.21.15.115:0/2529361835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26780051c0 msgr2=0x7f267800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:05.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26780051c0 0x7f267800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 10 monclient: _finish_hunting 0 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 monclient: found mon.a 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 -- 172.21.15.115:0/2529361835 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2678003570 con 0x7f2678004810 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 10 monclient: _finish_auth 0 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26922f3700 1 -- 172.21.15.115:0/2529361835 reap_dead start 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f26912f1700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 0x7f2678004c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f268800c2d0 tx=0x7f2688001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b17543144d40840 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:05.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268800cba0 con 0x7f2678004810 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: dump: 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:05.032 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:05.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268800cd00 con 0x7f2678004810 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 4 set_mon_vals no callback set 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 1 -- 172.21.15.115:0/2529361835 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688005b70 con 0x7f2678004810 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2690af0700 10 monclient: dump: 2023-09-30T01:06:05.034 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 10 monclient: shutdown 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 -- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 msgr2=0x7f2678004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 0x7f2678004c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f268800c2d0 tx=0x7f2688001e10 comp rx=0 tx=0).stop 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 -- 172.21.15.115:0/2529361835 shutdown_connections 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 --2- 172.21.15.115:0/2529361835 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2678004810 0x7f2678004c80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 -- 172.21.15.115:0/2529361835 >> 172.21.15.115:0/2529361835 conn(0x7f268c06a460 msgr2=0x7f268c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 -- 172.21.15.115:0/2529361835 shutdown_connections 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.997+0000 7f2693d56700 1 -- 172.21.15.115:0/2529361835 wait complete. 2023-09-30T01:06:05.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.998+0000 7f2693d56700 10 monclient: build_initial_monmap 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.998+0000 7f2693d56700 10 monclient: monmap: 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:05.037 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 librados: starting msgr at 2023-09-30T01:06:05.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 librados: starting objecter 2023-09-30T01:06:05.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 Processor -- start 2023-09-30T01:06:05.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 -- start start 2023-09-30T01:06:05.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 librados: setting wanted keys 2023-09-30T01:06:05.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 librados: calling monclient init 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 10 monclient: init 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:04.999+0000 7f2693d56700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-c con 0x7f2678004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 0x7f268c0a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-b con 0x7f268c0a9600 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f268c0a7cd0 0x7f268c14f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): picked mon.noname-a con 0x7f268c0a7cd0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0aa5b0 con 0x7f268c0a7cd0 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0abcd0 con 0x7f268c0a9600 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0a89e0 con 0x7f2678004810 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient(hunting): authenticate will time out at 1226.450073s 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f268c0a7cd0 0x7f268c14f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:37522/0 (socket says 172.21.15.115:37522) 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 -- 172.21.15.115:0/1396773361 learned_addr learned my addr 172.21.15.115:0/1396773361 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): get_auth_request con 0x7f2678004810 auth_method 0 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): get_auth_request con 0x7f268c0a7cd0 auth_method 0 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26922f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 0x7f268c0a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 10 monclient(hunting): get_auth_request con 0x7f268c0a9600 auth_method 0 2023-09-30T01:06:05.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:05.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:05.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:05.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2023-09-30T01:06:05.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 -- 172.21.15.115:0/1396773361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 msgr2=0x7f268c0a7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 0x7f268c0a7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 -- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f268c0a7cd0 msgr2=0x7f268c14f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f268c0a7cd0 0x7f268c14f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient: _finish_hunting 0 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 monclient: found mon.noname-c 2023-09-30T01:06:05.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 -- 172.21.15.115:0/1396773361 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0aa160 con 0x7f2678004810 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f26912f1700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 0x7f268c0a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2691af2700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f267c011a40 tx=0x7f267c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 client.?.objecter ms_handle_connect 0x7f2678004810 2023-09-30T01:06:05.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00f940 con 0x7f2678004810 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: dump: 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:05.048 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:05.049 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: _finish_auth 0 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00faa0 con 0x7f2678004810 2023-09-30T01:06:05.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 5 monclient: authenticate success, global_id 4508 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00fe20 con 0x7f2678004810 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient: _renew_subs 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c0a8150 con 0x7f2678004810 2023-09-30T01:06:05.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2681ffb700 10 monclient: dump: 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.052 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:05.054 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:05.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.000+0000 7f2683fff700 4 set_mon_vals no callback set 2023-09-30T01:06:05.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:05.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 monclient: _renew_subs 2023-09-30T01:06:05.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c14fb60 con 0x7f2678004810 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 1 librados: init done 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.001+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c0aa3b0 con 0x7f2678004810 2023-09-30T01:06:05.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f267c014650 con 0x7f2678004810 2023-09-30T01:06:05.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 1 --2- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 0x7f267006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:05.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f26912f1700 1 --2- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 0x7f267006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:05.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f267c018030 con 0x7f2678004810 2023-09-30T01:06:05.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f26912f1700 10 monclient: get_auth_request con 0x7f2670068890 auth_method 0 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 10 client.4508.objecter ms_dispatch 0x7f268c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 3 client.4508.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.002+0000 7f2681ffb700 3 client.4508.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.003+0000 7f2681ffb700 20 client.4508.objecter dump_active .. 0 homeless 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.003+0000 7f2681ffb700 10 client.4508.objecter ms_handle_connect 0x7f2670068890 2023-09-30T01:06:05.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.003+0000 7f26912f1700 1 --2- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 0x7f267006ad40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2688007c90 tx=0x7f2688015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:05.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.007+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f267c052030 con 0x7f2678004810 2023-09-30T01:06:05.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.007+0000 7f2681ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:05.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.007+0000 7f2681ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2693d56700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2693d56700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2693d56700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f268c072960 con 0x7f2678004810 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2681ffb700 1 -- 172.21.15.115:0/1396773361 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f267c055650 con 0x7f2678004810 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2681ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:05.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.230+0000 7f2681ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:05.232 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:05.232 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:05.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 10 librados: watch_flush enter 2023-09-30T01:06:05.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 10 librados: watch_flush exit 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 20 client.4508.objecter shutdown clearing up homeless session... 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 10 client.4508.objecter successfully canceled tick 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 msgr2=0x7f267006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 0x7f267006ad40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2688007c90 tx=0x7f2688015000 comp rx=0 tx=0).stop 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 10 monclient: shutdown 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 msgr2=0x7f268c13e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f267c011a40 tx=0x7f267c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 shutdown_connections 2023-09-30T01:06:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> v2:172.21.15.115:6816/115635 conn(0x7f2670068890 0x7f267006ad40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f268c0a7cd0 0x7f268c14f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2678004810 0x7f268c13e700 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 --2- 172.21.15.115:0/1396773361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f268c0a9600 0x7f268c0a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 >> 172.21.15.115:0/1396773361 conn(0x7f268c06a460 msgr2=0x7f268c1409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 shutdown_connections 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.232+0000 7f2693d56700 1 -- 172.21.15.115:0/1396773361 wait complete. 2023-09-30T01:06:05.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:05.233+0000 7f2693d56700 1 librados: shutdown 2023-09-30T01:06:08.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:08.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.374+0000 7fd9b278f700 10 monclient: get_monmap_and_config 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.374+0000 7fd9b278f700 10 monclient: build_initial_monmap 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.374+0000 7fd9b278f700 10 monclient: monmap: 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:08.375835+0000 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:08.375835+0000 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.375 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:08.376 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:08.376 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:08.376 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:08.376 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:08.376 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.375+0000 7fd9b278f700 1 Processor -- start 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.375+0000 7fd9b278f700 1 -- start start 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.375+0000 7fd9b278f700 10 monclient: init 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9ac0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 0x7fd9ac0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-b con 0x7fd9ac0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd9ac0a5c80 0x7fd9ac0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-c con 0x7fd9ac0a5c80 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0a7040 con 0x7fd9ac0ae8f0 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0a5480 con 0x7fd9ac0a5c80 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0a5760 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9b278f700 10 monclient(hunting): authenticate will time out at 1229.826380s 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 0x7fd9ac0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9abfff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd9ac0ae8f0 0x7fd9ac1415c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56676/0 (socket says 172.21.15.115:56676) 2023-09-30T01:06:08.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9abfff700 1 -- 172.21.15.115:0/58734066 learned_addr learned my addr 172.21.15.115:0/58734066 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request con 0x7fd9ac0aecc0 auth_method 0 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_done global_id 4511 payload 274 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9aaffd700 1 -- 172.21.15.115:0/58734066 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583016335 0 0) 0x7fd9ac0a7040 con 0x7fd9ac0ae8f0 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/58734066 >> v2:172.21.15.157:3300/0 conn(0x7fd9ac0a5c80 msgr2=0x7fd9ac0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/58734066 >> v2:172.21.15.157:3300/0 conn(0x7fd9ac0a5c80 0x7fd9ac0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/58734066 >> v1:172.21.15.115:6789/0 conn(0x7fd9ac0ae8f0 legacy=0x7fd9ac1415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 monclient: found mon.noname-b 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/58734066 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0b1970 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9aaffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.376+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/58734066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 0x7fd9ac0b1430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098e0 tx=0x7fd9a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b8414ebeaeb10d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 1 -- 172.21.15.115:0/58734066 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0009e80 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:08.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: dump: 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: _finish_auth 0 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 1 -- 172.21.15.115:0/58734066 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000b860 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 5 monclient: authenticate success, global_id 4511 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 1 -- 172.21.15.115:0/58734066 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000bbe0 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 4 set_mon_vals no callback set 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9aaffd700 10 monclient: dump: 2023-09-30T01:06:08.393 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 10 monclient: shutdown 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:08.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 -- 172.21.15.115:0/58734066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 msgr2=0x7fd9ac0b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 --2- 172.21.15.115:0/58734066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 0x7fd9ac0b1430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098e0 tx=0x7fd9a00091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 -- 172.21.15.115:0/58734066 shutdown_connections 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 --2- 172.21.15.115:0/58734066 >> v2:172.21.15.157:3300/0 conn(0x7fd9ac0a5c80 0x7fd9ac0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 --2- 172.21.15.115:0/58734066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0aecc0 0x7fd9ac0b1430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 -- 172.21.15.115:0/58734066 >> 172.21.15.115:0/58734066 conn(0x7fd9ac06a490 msgr2=0x7fd9ac0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 -- 172.21.15.115:0/58734066 shutdown_connections 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.377+0000 7fd9b278f700 1 -- 172.21.15.115:0/58734066 wait complete. 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 10 monclient: build_initial_monmap 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 10 monclient: monmap: 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:08.395 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 librados: starting msgr at 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 librados: starting objecter 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 Processor -- start 2023-09-30T01:06:08.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 -- start start 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 librados: setting wanted keys 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 1 librados: calling monclient init 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.378+0000 7fd9b278f700 10 monclient: init 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-c con 0x7fd9ac0a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd9ac14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-a con 0x7fd9ac0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd9ac0a9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): picked mon.noname-b con 0x7fd9ac0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac14e170 con 0x7fd9ac0ae8f0 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0aafd0 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0a7cc0 con 0x7fd9ac0a5c80 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b278f700 10 monclient(hunting): authenticate will time out at 1229.829434s 2023-09-30T01:06:08.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd9ac14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd9ac14eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:38718/0 (socket says 172.21.15.115:38718) 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:37540/0 (socket says 172.21.15.115:37540) 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/660783143 learned_addr learned my addr 172.21.15.115:0/660783143 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request con 0x7fd9ac0ae8f0 auth_method 0 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): get_auth_request con 0x7fd9ac0a5c80 auth_method 0 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient(hunting): handle_auth_done global_id 4514 payload 824 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b0d2c700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd9ac0a9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 msgr2=0x7fd9ac0a9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd9ac0a9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 msgr2=0x7fd9ac14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd9ac14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 monclient: found mon.noname-c 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0009420 con 0x7fd9ac0a5c80 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b0d2c700 10 monclient: get_auth_request con 0x7fd9ac0aecc0 auth_method 0 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9b0d2c700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd9ac0a9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.379+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c5d0 tx=0x7fd99c007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 client.?.objecter ms_handle_connect 0x7fd9ac0a5c80 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:08.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 msgr2=0x7fd9ac14e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 msgr2=0x7fd9ac14e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c5d0 tx=0x7fd99c007d30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c5d0 tx=0x7fd99c007d30 comp rx=0 tx=0).stop 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 msgr2=0x7fd9ac14e600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd9ac0a5c80 0x7fd9ac14e600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd998006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd9ac0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 0x7fd998009d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd998007880 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd99800c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd9ac0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99800d1b0 con 0x7fd9ac0ae8f0 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99800d320 con 0x7fd9ac0aecc0 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99800d4e0 con 0x7fd998007880 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 0x7fd998009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request con 0x7fd998007880 auth_method 0 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd99800c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): get_auth_request con 0x7fd9ac0ae8f0 auth_method 0 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd998006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b0d2c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 10 monclient(hunting): get_auth_request con 0x7fd9ac0aecc0 auth_method 0 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:08.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9abfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient(hunting): handle_auth_done global_id 4514 payload 293 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 msgr2=0x7fd998006ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd998006ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 msgr2=0x7fd99800c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd99800c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 monclient: found mon.noname-c 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99800d700 con 0x7fd998007880 2023-09-30T01:06:08.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 client.?.objecter ms_handle_connect 0x7fd998007880 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9ab7fe700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 0x7fd998009d00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac13f9c0 tx=0x7fd9a0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0022070 con 0x7fd998007880 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: dump: 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:08.407 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: _finish_auth 0 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000bbc0 con 0x7fd998007880 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a001a450 con 0x7fd998007880 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 5 monclient: authenticate success, global_id 4514 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 10 monclient: _renew_subs 2023-09-30T01:06:08.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac13d810 con 0x7fd998007880 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9aa7fc700 4 set_mon_vals no callback set 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd993fff700 10 monclient: dump: 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:08.409 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 10 monclient: _renew_subs 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac14e7c0 con 0x7fd998007880 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.380+0000 7fd9b278f700 1 librados: init done 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.381+0000 7fd9b278f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.381+0000 7fd9b278f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:08.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.381+0000 7fd9b278f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.381+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac14e320 con 0x7fd998007880 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd9a001a880 con 0x7fd998007880 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 1 --2- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 0x7fd998072d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 0x7fd998072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd9a0084620 con 0x7fd998007880 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 10 client.4514.objecter ms_dispatch 0x7fd9ac001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 3 client.4514.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 3 client.4514.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd9abfff700 10 monclient: get_auth_request con 0x7fd9ac0a5c80 auth_method 0 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 20 client.4514.objecter dump_active .. 0 homeless 2023-09-30T01:06:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd993fff700 10 client.4514.objecter ms_handle_connect 0x7fd9ac0a5c80 2023-09-30T01:06:08.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.382+0000 7fd9abfff700 1 --2- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 0x7fd998072d30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007c40 tx=0x7fd99c015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:08.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.386+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9ac14e320 con 0x7fd998007880 2023-09-30T01:06:08.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.386+0000 7fd993fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:08.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.386+0000 7fd993fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:08.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd9b278f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd9b278f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd9b278f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd9ac14e320 con 0x7fd998007880 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd993fff700 1 -- 172.21.15.115:0/660783143 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd9ac14e320 con 0x7fd998007880 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd993fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:08.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.609+0000 7fd993fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:08.611 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:08.611 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:08.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.611+0000 7fd9b278f700 10 librados: watch_flush enter 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.611+0000 7fd9b278f700 10 librados: watch_flush exit 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.611+0000 7fd9b278f700 20 client.4514.objecter shutdown clearing up homeless session... 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 10 client.4514.objecter successfully canceled tick 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 msgr2=0x7fd998072d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 0x7fd998072d30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007c40 tx=0x7fd99c015000 comp rx=0 tx=0).stop 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 10 monclient: shutdown 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 msgr2=0x7fd998009d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 0x7fd998009d00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac13f9c0 tx=0x7fd9a0017000 comp rx=0 tx=0).stop 2023-09-30T01:06:08.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 shutdown_connections 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> v2:172.21.15.115:6816/115635 conn(0x7fd9ac0a5c80 0x7fd998072d30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd9ac0ae8f0 0x7fd99800c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd998007880 0x7fd998009d00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 --2- 172.21.15.115:0/660783143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9ac0aecc0 0x7fd998006ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 >> 172.21.15.115:0/660783143 conn(0x7fd9ac06a490 msgr2=0x7fd9ac0b5150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 shutdown_connections 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 -- 172.21.15.115:0/660783143 wait complete. 2023-09-30T01:06:08.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:08.612+0000 7fd9b278f700 1 librados: shutdown 2023-09-30T01:06:11.625 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:11.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.753+0000 7fef4fcbe700 10 monclient: get_monmap_and_config 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.753+0000 7fef4fcbe700 10 monclient: build_initial_monmap 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.753+0000 7fef4fcbe700 10 monclient: monmap: 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:11.754857+0000 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:11.754857+0000 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.754 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:11.755 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:11.755 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:11.755 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:11.755 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:11.755 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.754+0000 7fef4fcbe700 1 Processor -- start 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.754+0000 7fef4fcbe700 1 -- start start 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.754+0000 7fef4fcbe700 10 monclient: init 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 0x7fef480a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:11.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-b con 0x7fef480ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef480aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef481415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480a7040 con 0x7fef480aecc0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480a57f0 con 0x7fef481415c0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef480a5480 con 0x7fef480ae8f0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4fcbe700 10 monclient(hunting): authenticate will time out at 1233.205336s 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 0x7fef480a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4d259700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fef480aecc0 0x7fef480b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:48122/0 (socket says 172.21.15.115:48122) 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4d259700 1 -- 172.21.15.115:0/1215515030 learned_addr learned my addr 172.21.15.115:0/1215515030 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): get_auth_request con 0x7fef480ae8f0 auth_method 0 2023-09-30T01:06:11.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4ca58700 1 -- 172.21.15.115:0/1215515030 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741440964 0 0) 0x7fef480a7040 con 0x7fef480aecc0 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4ca58700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4ca58700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4ca58700 10 monclient(hunting): my global_id is 4695 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4ca58700 1 -- 172.21.15.115:0/1215515030 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef34003570 con 0x7fef480aecc0 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient(hunting): handle_auth_done global_id 4517 payload 274 2023-09-30T01:06:11.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 -- 172.21.15.115:0/1215515030 >> v1:172.21.15.157:6789/0 conn(0x7fef481415c0 legacy=0x7fef480adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 -- 172.21.15.115:0/1215515030 >> v1:172.21.15.115:6789/0 conn(0x7fef480aecc0 legacy=0x7fef480b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient: _finish_hunting 0 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 monclient: found mon.noname-b 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 -- 172.21.15.115:0/1215515030 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480a55c0 con 0x7fef480ae8f0 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.755+0000 7fef4da5a700 1 --2- 172.21.15.115:0/1215515030 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 0x7fef480a5d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fef38009910 tx=0x7fef380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4bade381dbe4ee2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 1 -- 172.21.15.115:0/1215515030 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3800b640 con 0x7fef480ae8f0 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: dump: 2023-09-30T01:06:11.766 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: _finish_auth 0 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:11.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 1 -- 172.21.15.115:0/1215515030 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3800b7a0 con 0x7fef480ae8f0 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 5 monclient: authenticate success, global_id 4517 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 1 -- 172.21.15.115:0/1215515030 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3800bb20 con 0x7fef480ae8f0 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 4 set_mon_vals no callback set 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4ca58700 10 monclient: dump: 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:11.768 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 10 monclient: shutdown 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 -- 172.21.15.115:0/1215515030 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 msgr2=0x7fef480a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/1215515030 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 0x7fef480a5d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fef38009910 tx=0x7fef380091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 -- 172.21.15.115:0/1215515030 shutdown_connections 2023-09-30T01:06:11.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/1215515030 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480ae8f0 0x7fef480a5d10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 -- 172.21.15.115:0/1215515030 >> 172.21.15.115:0/1215515030 conn(0x7fef4806a460 msgr2=0x7fef480630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 -- 172.21.15.115:0/1215515030 shutdown_connections 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.756+0000 7fef4fcbe700 1 -- 172.21.15.115:0/1215515030 wait complete. 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.757+0000 7fef4fcbe700 10 monclient: build_initial_monmap 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.757+0000 7fef4fcbe700 10 monclient: monmap: 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.770 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.757+0000 7fef4fcbe700 1 librados: starting msgr at 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.757+0000 7fef4fcbe700 1 librados: starting objecter 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 Processor -- start 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 -- start start 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 librados: setting wanted keys 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 librados: calling monclient init 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient: init 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:11.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-a con 0x7fef480ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480aecc0 0x7fef4814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-c con 0x7fef480aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef481415c0 0x7fef4814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): picked mon.noname-b con 0x7fef481415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48151eb0 con 0x7fef480ae8f0 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480aba80 con 0x7fef481415c0 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:11.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef48151bb0 con 0x7fef480aecc0 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4fcbe700 10 monclient(hunting): authenticate will time out at 1233.208927s 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:58214/0 (socket says 172.21.15.115:58214) 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 1 -- 172.21.15.115:0/2834683264 learned_addr learned my addr 172.21.15.115:0/2834683264 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 10 monclient(hunting): get_auth_request con 0x7fef480ae8f0 auth_method 0 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.758+0000 7fef4da5a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480aecc0 0x7fef4814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:11.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): get_auth_request con 0x7fef480aecc0 auth_method 0 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4e25b700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef481415c0 0x7fef4814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4d259700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4e25b700 10 monclient(hunting): get_auth_request con 0x7fef481415c0 auth_method 0 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4e25b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:11.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4e25b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4e25b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient(hunting): handle_auth_done global_id 4698 payload 824 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 -- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480aecc0 msgr2=0x7fef4814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480aecc0 0x7fef4814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 -- 172.21.15.115:0/2834683264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef481415c0 msgr2=0x7fef4814e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef481415c0 0x7fef4814e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient: _finish_hunting 0 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 monclient: found mon.noname-a 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:11.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 -- 172.21.15.115:0/2834683264 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef38009420 con 0x7fef480ae8f0 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4da5a700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef38009fb0 tx=0x7fef38016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 client.?.objecter ms_handle_connect 0x7fef480ae8f0 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3800bb20 con 0x7fef480ae8f0 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: dump: 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:11.777 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: _finish_auth 0 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 5 monclient: authenticate success, global_id 4698 2023-09-30T01:06:11.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef38000cc0 con 0x7fef480ae8f0 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 10 monclient: _renew_subs 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef480a87e0 con 0x7fef480ae8f0 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3801c680 con 0x7fef480ae8f0 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3dffb700 10 monclient: dump: 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:11.779 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef3ffff700 4 set_mon_vals no callback set 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 10 monclient: _renew_subs 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4814ea30 con 0x7fef480ae8f0 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.759+0000 7fef4fcbe700 1 librados: init done 2023-09-30T01:06:11.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.760+0000 7fef4fcbe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.760+0000 7fef4fcbe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.760+0000 7fef4fcbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.760+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48072960 con 0x7fef480ae8f0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fef3801ca30 con 0x7fef480ae8f0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 1 --2- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 0x7fef2c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fef380857e0 con 0x7fef480ae8f0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 10 client.4698.objecter ms_dispatch 0x7fef48001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 3 client.4698.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 3 client.4698.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef4d259700 1 --2- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 0x7fef2c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef4d259700 10 monclient: get_auth_request con 0x7fef2c068890 auth_method 0 2023-09-30T01:06:11.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 20 client.4698.objecter dump_active .. 0 homeless 2023-09-30T01:06:11.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef3dffb700 10 client.4698.objecter ms_handle_connect 0x7fef2c068890 2023-09-30T01:06:11.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.761+0000 7fef4d259700 1 --2- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 0x7fef2c06ad40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef4813eee0 tx=0x7fef44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:11.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.766+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef380530a0 con 0x7fef480ae8f0 2023-09-30T01:06:11.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.766+0000 7fef3dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:11.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.766+0000 7fef3dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:11.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef4fcbe700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef4fcbe700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef4fcbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fef4814ea30 con 0x7fef480ae8f0 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef3dffb700 1 -- 172.21.15.115:0/2834683264 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fef4814ea30 con 0x7fef480ae8f0 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef3dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.984+0000 7fef3dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:11.985 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 10 librados: watch_flush enter 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 10 librados: watch_flush exit 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 20 client.4698.objecter shutdown clearing up homeless session... 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 10 client.4698.objecter successfully canceled tick 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 msgr2=0x7fef2c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 0x7fef2c06ad40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef4813eee0 tx=0x7fef44009040 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 10 monclient: shutdown 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 msgr2=0x7fef48151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef38009fb0 tx=0x7fef38016000 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 shutdown_connections 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> v2:172.21.15.115:6816/115635 conn(0x7fef2c068890 0x7fef2c06ad40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef481415c0 0x7fef4814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fef480ae8f0 0x7fef48151670 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 --2- 172.21.15.115:0/2834683264 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fef480aecc0 0x7fef4814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.986+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 >> 172.21.15.115:0/2834683264 conn(0x7fef4806a460 msgr2=0x7fef480630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.987+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 shutdown_connections 2023-09-30T01:06:11.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.987+0000 7fef4fcbe700 1 -- 172.21.15.115:0/2834683264 wait complete. 2023-09-30T01:06:11.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:11.987+0000 7fef4fcbe700 1 librados: shutdown 2023-09-30T01:06:14.999 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:15.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.126+0000 7faa6ef42700 10 monclient: get_monmap_and_config 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.126+0000 7faa6ef42700 10 monclient: build_initial_monmap 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.126+0000 7faa6ef42700 10 monclient: monmap: 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:15.127868+0000 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:15.127868+0000 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.127 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:15.128 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:15.128 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:15.128 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:15.128 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:15.128 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.127+0000 7faa6ef42700 1 Processor -- start 2023-09-30T01:06:15.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.127+0000 7faa6ef42700 1 -- start start 2023-09-30T01:06:15.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.127+0000 7faa6ef42700 10 monclient: init 2023-09-30T01:06:15.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faa680ae8f0 0x7faa680a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-c con 0x7faa680ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-b con 0x7faa680aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-a con 0x7faa681415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa680a7040 con 0x7faa681415c0 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa680b19b0 con 0x7faa680ae8f0 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680a57f0 con 0x7faa680aecc0 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ef42700 10 monclient(hunting): authenticate will time out at 1236.578334s 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:58228/0 (socket says 172.21.15.115:58228) 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 1 -- 172.21.15.115:0/3383255913 learned_addr learned my addr 172.21.15.115:0/3383255913 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50596/0 (socket says 172.21.15.115:50596) 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): get_auth_request con 0x7faa681415c0 auth_method 0 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:15.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): get_auth_request con 0x7faa680aecc0 auth_method 0 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:15.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 -- 172.21.15.115:0/3383255913 >> v2:172.21.15.157:3300/0 conn(0x7faa680ae8f0 msgr2=0x7faa680a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6ccde700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.157:3300/0 conn(0x7faa680ae8f0 0x7faa680a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.157:3300/0 conn(0x7faa680ae8f0 0x7faa680a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 -- 172.21.15.115:0/3383255913 >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 msgr2=0x7faa680adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 monclient: found mon.noname-b 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 -- 172.21.15.115:0/3383255913 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680a5480 con 0x7faa680aecc0 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa6d4df700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:15.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.128+0000 7faa67fff700 1 --2- 172.21.15.115:0/3383255913 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faa680b59c0 tx=0x7faa540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6281370398825ecc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 1 -- 172.21.15.115:0/3383255913 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54011630 con 0x7faa680aecc0 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: dump: 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:15.142 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: _finish_auth 0 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 1 -- 172.21.15.115:0/3383255913 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa54011790 con 0x7faa680aecc0 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 5 monclient: authenticate success, global_id 4523 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 1 -- 172.21.15.115:0/3383255913 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54011b30 con 0x7faa680aecc0 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:15.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa677fe700 10 monclient: dump: 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:15.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 4 set_mon_vals no callback set 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 10 monclient: shutdown 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 -- 172.21.15.115:0/3383255913 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 msgr2=0x7faa680b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 --2- 172.21.15.115:0/3383255913 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faa680b59c0 tx=0x7faa540091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 -- 172.21.15.115:0/3383255913 shutdown_connections 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.115:3300/0 conn(0x7faa681415c0 0x7faa680adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 --2- 172.21.15.115:0/3383255913 >> v2:172.21.15.157:3300/0 conn(0x7faa680ae8f0 0x7faa680a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 --2- 172.21.15.115:0/3383255913 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa680b1470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 -- 172.21.15.115:0/3383255913 >> 172.21.15.115:0/3383255913 conn(0x7faa6806a460 msgr2=0x7faa680630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 -- 172.21.15.115:0/3383255913 shutdown_connections 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.129+0000 7faa6ef42700 1 -- 172.21.15.115:0/3383255913 wait complete. 2023-09-30T01:06:15.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.130+0000 7faa6ef42700 10 monclient: build_initial_monmap 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.130+0000 7faa6ef42700 10 monclient: monmap: 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.130+0000 7faa6ef42700 1 librados: starting msgr at 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.130+0000 7faa6ef42700 1 librados: starting objecter 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.130+0000 7faa6ef42700 1 Processor -- start 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 -- start start 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 librados: setting wanted keys 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 librados: calling monclient init 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient: init 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-a con 0x7faa680ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:15.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa68151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-c con 0x7faa680aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa681415c0 0x7faa680a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): picked mon.noname-b con 0x7faa681415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68151c20 con 0x7faa680ae8f0 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680ab5c0 con 0x7faa681415c0 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa681517a0 con 0x7faa680aecc0 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ef42700 10 monclient(hunting): authenticate will time out at 1236.581670s 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:58240/0 (socket says 172.21.15.115:58240) 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 1 -- 172.21.15.115:0/2407181329 learned_addr learned my addr 172.21.15.115:0/2407181329 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): get_auth_request con 0x7faa680ae8f0 auth_method 0 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa68151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): get_auth_request con 0x7faa680aecc0 auth_method 0 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:15.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6ccde700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6d4df700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa681415c0 0x7faa680a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6d4df700 10 monclient(hunting): get_auth_request con 0x7faa681415c0 auth_method 0 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6d4df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6d4df700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.131+0000 7faa6d4df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:15.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6d4df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 10 monclient(hunting): handle_auth_done global_id 4704 payload 824 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 -- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 msgr2=0x7faa68151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa68151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 -- 172.21.15.115:0/2407181329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa681415c0 msgr2=0x7faa680a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa681415c0 0x7faa680a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 10 monclient: _finish_hunting 0 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 monclient: found mon.noname-a 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 -- 172.21.15.115:0/2407181329 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60009ae0 con 0x7faa680ae8f0 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa67fff700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa68151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ccde700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00c7d0 tx=0x7faa5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 client.?.objecter ms_handle_connect 0x7faa680ae8f0 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c011600 con 0x7faa680ae8f0 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: dump: 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.153 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: _finish_auth 0 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c011760 con 0x7faa680ae8f0 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 5 monclient: authenticate success, global_id 4704 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 10 monclient: _renew_subs 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa54009420 con 0x7faa680ae8f0 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:15.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c011b20 con 0x7faa680ae8f0 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa64ff9700 10 monclient: dump: 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:15.155 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa66ffd700 4 set_mon_vals no callback set 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 10 client.4704.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 10 monclient: _renew_subs 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa680a9b70 con 0x7faa680ae8f0 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.132+0000 7faa6ef42700 1 librados: init done 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.133+0000 7faa6ef42700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.133+0000 7faa6ef42700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.133+0000 7faa6ef42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.133+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa68072960 con 0x7faa680ae8f0 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.133+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7faa5c0193f0 con 0x7faa680ae8f0 2023-09-30T01:06:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 1 --2- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 0x7faa4c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa67fff700 1 --2- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 0x7faa4c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa67fff700 10 monclient: get_auth_request con 0x7faa4c068890 auth_method 0 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7faa5c084220 con 0x7faa680ae8f0 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 10 client.4704.objecter ms_dispatch 0x7faa68001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 3 client.4704.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 3 client.4704.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 20 client.4704.objecter dump_active .. 0 homeless 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa64ff9700 10 client.4704.objecter ms_handle_connect 0x7faa4c068890 2023-09-30T01:06:15.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.134+0000 7faa67fff700 1 --2- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 0x7faa4c06ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa540093f0 tx=0x7faa54017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:15.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.138+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa5c051ae0 con 0x7faa680ae8f0 2023-09-30T01:06:15.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.138+0000 7faa64ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:15.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.138+0000 7faa64ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:15.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa6ef42700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:15.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa6ef42700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa6ef42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faa6806f840 con 0x7faa680ae8f0 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa64ff9700 1 -- 172.21.15.115:0/2407181329 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7faa5c055100 con 0x7faa680ae8f0 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa64ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.358+0000 7faa64ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:15.359 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 10 librados: watch_flush enter 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 10 librados: watch_flush exit 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 20 client.4704.objecter shutdown clearing up homeless session... 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 10 client.4704.objecter successfully canceled tick 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 msgr2=0x7faa4c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 0x7faa4c06ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa540093f0 tx=0x7faa54017000 comp rx=0 tx=0).stop 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 10 monclient: shutdown 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:15.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 msgr2=0x7faa6813e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00c7d0 tx=0x7faa5c008040 comp rx=0 tx=0).stop 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 shutdown_connections 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> v2:172.21.15.115:6816/115635 conn(0x7faa4c068890 0x7faa4c06ad40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa681415c0 0x7faa680a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa680ae8f0 0x7faa6813e690 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 --2- 172.21.15.115:0/2407181329 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7faa680aecc0 0x7faa68151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 >> 172.21.15.115:0/2407181329 conn(0x7faa6806a460 msgr2=0x7faa6813abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 shutdown_connections 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 -- 172.21.15.115:0/2407181329 wait complete. 2023-09-30T01:06:15.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:15.360+0000 7faa6ef42700 1 librados: shutdown 2023-09-30T01:06:18.373 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:18.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.501+0000 7f1784be3700 10 monclient: get_monmap_and_config 2023-09-30T01:06:18.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.501+0000 7f1784be3700 10 monclient: build_initial_monmap 2023-09-30T01:06:18.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.501+0000 7f1784be3700 10 monclient: monmap: 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:18.502530+0000 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:18.502530+0000 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:18.502 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:18.503 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:18.503 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 Processor -- start 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 -- start start 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient: init 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17800ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17800aecc0 0x7f17800a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-c con 0x7f17800aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-b con 0x7f17801415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:18.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17800a7040 con 0x7f17800ae8f0 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17800a57f0 con 0x7f17800aecc0 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17800a5480 con 0x7f17801415c0 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.502+0000 7f1784be3700 10 monclient(hunting): authenticate will time out at 1239.952925s 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50608/0 (socket says 172.21.15.115:50608) 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 -- 172.21.15.115:0/2992778038 learned_addr learned my addr 172.21.15.115:0/2992778038 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): get_auth_request con 0x7f17801415c0 auth_method 0 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:18.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177dd9b700 1 --2- 172.21.15.115:0/2992778038 >> v2:172.21.15.157:3300/0 conn(0x7f17800aecc0 0x7f17800a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 -- 172.21.15.115:0/2992778038 >> v2:172.21.15.157:3300/0 conn(0x7f17800aecc0 msgr2=0x7f17800a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 --2- 172.21.15.115:0/2992778038 >> v2:172.21.15.157:3300/0 conn(0x7f17800aecc0 0x7f17800a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 -- 172.21.15.115:0/2992778038 >> v1:172.21.15.157:6789/0 conn(0x7f17800ae8f0 legacy=0x7f17800b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:06:18.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 monclient: found mon.noname-b 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 -- 172.21.15.115:0/2992778038 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780141a40 con 0x7f17801415c0 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177ed9d700 1 --2- 172.21.15.115:0/2992778038 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1778009f20 tx=0x7f1778009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5579ff2322d40764 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 1 -- 172.21.15.115:0/2992778038 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177800e070 con 0x7f17801415c0 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: dump: 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:18.514 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: _finish_auth 0 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 1 -- 172.21.15.115:0/2992778038 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177800ba40 con 0x7f17801415c0 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f1784be3700 5 monclient: authenticate success, global_id 4529 2023-09-30T01:06:18.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f1784be3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f1784be3700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 1 -- 172.21.15.115:0/2992778038 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17780092e0 con 0x7f17801415c0 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f1784be3700 4 set_mon_vals no callback set 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.503+0000 7f177d59a700 10 monclient: dump: 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.516 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 10 monclient: shutdown 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 -- 172.21.15.115:0/2992778038 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 msgr2=0x7f17800adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:18.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 --2- 172.21.15.115:0/2992778038 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1778009f20 tx=0x7f1778009840 comp rx=0 tx=0).stop 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 -- 172.21.15.115:0/2992778038 shutdown_connections 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 --2- 172.21.15.115:0/2992778038 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17801415c0 0x7f17800adb60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 --2- 172.21.15.115:0/2992778038 >> v2:172.21.15.157:3300/0 conn(0x7f17800aecc0 0x7f17800a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 -- 172.21.15.115:0/2992778038 >> 172.21.15.115:0/2992778038 conn(0x7f178006a460 msgr2=0x7f17800630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 -- 172.21.15.115:0/2992778038 shutdown_connections 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 1 -- 172.21.15.115:0/2992778038 wait complete. 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 10 monclient: build_initial_monmap 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.504+0000 7f1784be3700 10 monclient: monmap: 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:18.518 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 librados: starting msgr at 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 librados: starting objecter 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 Processor -- start 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 -- start start 2023-09-30T01:06:18.519 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 librados: setting wanted keys 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 librados: calling monclient init 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient: init 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-c con 0x7f17800ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-a con 0x7f17800aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17801415c0 0x7f17800a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): picked mon.noname-b con 0x7f17801415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:18.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17800a77b0 con 0x7f17800aecc0 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17800ad9e0 con 0x7f17801415c0 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178014e4b0 con 0x7f17800ae8f0 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f1784be3700 10 monclient(hunting): authenticate will time out at 1239.955691s 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50614/0 (socket says 172.21.15.115:50614) 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 1 -- 172.21.15.115:0/702505400 learned_addr learned my addr 172.21.15.115:0/702505400 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:58242/0 (socket says 172.21.15.115:58242) 2023-09-30T01:06:18.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): get_auth_request con 0x7f17800aecc0 auth_method 0 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): get_auth_request con 0x7f17800ae8f0 auth_method 0 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:18.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.505+0000 7f177ed9d700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17801415c0 0x7f17800a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177ed9d700 10 monclient(hunting): get_auth_request con 0x7f17801415c0 auth_method 0 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 -- 172.21.15.115:0/702505400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17801415c0 msgr2=0x7f17800a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17801415c0 0x7f17800a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 -- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 msgr2=0x7f178014df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:18.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 monclient: found mon.noname-c 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 -- 172.21.15.115:0/702505400 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778009ae0 con 0x7f17800ae8f0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177dd9b700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177e59c700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f177000c5d0 tx=0x7f1770007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 client.?.objecter ms_handle_connect 0x7f17800ae8f0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770011590 con 0x7f17800ae8f0 2023-09-30T01:06:18.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: dump: 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:18.525 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770011710 con 0x7f17800ae8f0 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 5 monclient: authenticate success, global_id 4532 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 10 monclient: _renew_subs 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178014e6a0 con 0x7f17800ae8f0 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770011ad0 con 0x7f17800ae8f0 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:18.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f176e7fc700 10 monclient: dump: 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f177cd99700 4 set_mon_vals no callback set 2023-09-30T01:06:18.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 10 monclient: _renew_subs 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17800a8380 con 0x7f17800ae8f0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.506+0000 7f1784be3700 1 librados: init done 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.507+0000 7f1784be3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.507+0000 7f1784be3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.507+0000 7f1784be3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.507+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17800a7920 con 0x7f17800ae8f0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.507+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f177001c460 con 0x7f17800ae8f0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 1 --2- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 0x7f176006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f177dd9b700 1 --2- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 0x7f176006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f177dd9b700 10 monclient: get_auth_request con 0x7f1760068890 auth_method 0 2023-09-30T01:06:18.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1770083bc0 con 0x7f17800ae8f0 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 10 client.4532.objecter ms_dispatch 0x7f1780001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 3 client.4532.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 3 client.4532.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 20 client.4532.objecter dump_active .. 0 homeless 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f176e7fc700 10 client.4532.objecter ms_handle_connect 0x7f1760068890 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.508+0000 7f177dd9b700 1 --2- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 0x7f176006ad40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f177400f360 tx=0x7f1774008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:18.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.512+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17800a7920 con 0x7f17800ae8f0 2023-09-30T01:06:18.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.512+0000 7f176e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:18.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.512+0000 7f176e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.734+0000 7f1784be3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.734+0000 7f1784be3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.734+0000 7f1784be3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.734+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17800a7920 con 0x7f17800ae8f0 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.735+0000 7f176e7fc700 1 -- 172.21.15.115:0/702505400 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f17800a7920 con 0x7f17800ae8f0 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.735+0000 7f176e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.735+0000 7f176e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:18.736 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:18.736 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.736+0000 7f1784be3700 10 librados: watch_flush enter 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.736+0000 7f1784be3700 10 librados: watch_flush exit 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.736+0000 7f1784be3700 20 client.4532.objecter shutdown clearing up homeless session... 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.736+0000 7f1784be3700 10 client.4532.objecter successfully canceled tick 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 msgr2=0x7f176006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:18.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 0x7f176006ad40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f177400f360 tx=0x7f1774008040 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 10 monclient: shutdown 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 msgr2=0x7f178013e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f177000c5d0 tx=0x7f1770007f60 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 shutdown_connections 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> v2:172.21.15.115:6816/115635 conn(0x7f1760068890 0x7f176006ad40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17801415c0 0x7f17800a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f17800ae8f0 0x7f178013e4d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 --2- 172.21.15.115:0/702505400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f17800aecc0 0x7f178014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 >> 172.21.15.115:0/702505400 conn(0x7f178006a460 msgr2=0x7f178013fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 shutdown_connections 2023-09-30T01:06:18.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 -- 172.21.15.115:0/702505400 wait complete. 2023-09-30T01:06:18.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:18.737+0000 7f1784be3700 1 librados: shutdown 2023-09-30T01:06:21.749 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:21.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.878+0000 7fd84b9a5700 10 monclient: get_monmap_and_config 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.878+0000 7fd84b9a5700 10 monclient: build_initial_monmap 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.878+0000 7fd84b9a5700 10 monclient: monmap: 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:21.879664+0000 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:21.879664+0000 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:21.880 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:21.880 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.879+0000 7fd84b9a5700 1 Processor -- start 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.879+0000 7fd84b9a5700 1 -- start start 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.879+0000 7fd84b9a5700 10 monclient: init 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8440ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd8440aecc0 0x7fd8440a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-c con 0x7fd8440aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8441415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:21.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440a7040 con 0x7fd8441415c0 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440a57f0 con 0x7fd8440ae8f0 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8440a5480 con 0x7fd8440aecc0 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd84b9a5700 10 monclient(hunting): authenticate will time out at 1243.330198s 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd849f42700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd8441415c0 0x7fd8440adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:60500/0 (socket says 172.21.15.115:60500) 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd849f42700 1 -- 172.21.15.115:0/925586207 learned_addr learned my addr 172.21.15.115:0/925586207 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 1 --2- 172.21.15.115:0/925586207 >> v2:172.21.15.157:3300/0 conn(0x7fd8440aecc0 0x7fd8440a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): get_auth_request con 0x7fd8440aecc0 auth_method 0 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:21.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322371854 0 0) 0x7fd8440a7040 con 0x7fd8441415c0 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): my global_id is 4710 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd830003570 con 0x7fd8441415c0 2023-09-30T01:06:21.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586834672 0 0) 0x7fd8440a57f0 con 0x7fd8440ae8f0 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient(hunting): my global_id is 4615 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8440a7040 con 0x7fd8440ae8f0 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient(hunting): handle_auth_done global_id 4612 payload 274 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 1 -- 172.21.15.115:0/925586207 >> v1:172.21.15.157:6789/0 conn(0x7fd8440ae8f0 legacy=0x7fd8440b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 1 -- 172.21.15.115:0/925586207 >> v1:172.21.15.115:6789/0 conn(0x7fd8441415c0 legacy=0x7fd8440adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3060178904 0 0) 0x7fd830003570 con 0x7fd8441415c0 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient: _finish_hunting 0 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 1 monclient: found mon.noname-c 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:06:21.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd848f40700 1 -- 172.21.15.115:0/925586207 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440a55c0 con 0x7fd8440aecc0 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.880+0000 7fd83bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd848f40700 1 --2- 172.21.15.115:0/925586207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd8440a5d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd8340098e0 tx=0x7fd8340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17854a7c81a59744 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0017f0 con 0x7fd8440aecc0 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: dump: 2023-09-30T01:06:21.893 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.894 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: _finish_auth 0 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:21.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd834011420 con 0x7fd8440aecc0 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 5 monclient: authenticate success, global_id 4612 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 1 -- 172.21.15.115:0/925586207 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8340117c0 con 0x7fd8440aecc0 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 4 set_mon_vals no callback set 2023-09-30T01:06:21.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd83bfff700 10 monclient: dump: 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.897 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 10 monclient: shutdown 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 -- 172.21.15.115:0/925586207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 msgr2=0x7fd8440a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/925586207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd8440a5d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd8340098e0 tx=0x7fd8340091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 -- 172.21.15.115:0/925586207 shutdown_connections 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/925586207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd8440a5d10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 -- 172.21.15.115:0/925586207 >> 172.21.15.115:0/925586207 conn(0x7fd84406a460 msgr2=0x7fd8440630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 -- 172.21.15.115:0/925586207 shutdown_connections 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.881+0000 7fd84b9a5700 1 -- 172.21.15.115:0/925586207 wait complete. 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 10 monclient: build_initial_monmap 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 10 monclient: monmap: 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:21.899 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:21.900 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 librados: starting msgr at 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 librados: starting objecter 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 Processor -- start 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 -- start start 2023-09-30T01:06:21.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 librados: setting wanted keys 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 1 librados: calling monclient init 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.882+0000 7fd84b9a5700 10 monclient: init 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:21.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-a con 0x7fd8440ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd844150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-b con 0x7fd8440aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8441415c0 0x7fd8440a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): picked mon.noname-c con 0x7fd8441415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:21.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84413a920 con 0x7fd8440ae8f0 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440a99e0 con 0x7fd8440aecc0 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84413ecb0 con 0x7fd8441415c0 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd84b9a5700 10 monclient(hunting): authenticate will time out at 1243.333559s 2023-09-30T01:06:21.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59866/0 (socket says 172.21.15.115:59866) 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 -- 172.21.15.115:0/3781348357 learned_addr learned my addr 172.21.15.115:0/3781348357 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): get_auth_request con 0x7fd8440ae8f0 auth_method 0 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd844150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8441415c0 0x7fd8440a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): get_auth_request con 0x7fd8440aecc0 auth_method 0 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): get_auth_request con 0x7fd8441415c0 auth_method 0 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:21.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd848f40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849f42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:21.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient(hunting): handle_auth_done global_id 4713 payload 824 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 -- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8441415c0 msgr2=0x7fd8440a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8441415c0 0x7fd8440a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 -- 172.21.15.115:0/3781348357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 msgr2=0x7fd844150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd844150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient: _finish_hunting 0 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 monclient: found mon.noname-a 2023-09-30T01:06:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.883+0000 7fd849741700 1 -- 172.21.15.115:0/3781348357 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834009420 con 0x7fd8440ae8f0 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd849741700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd84000c580 tx=0x7fd840007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 client.?.objecter ms_handle_connect 0x7fd8440ae8f0 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:21.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840010510 con 0x7fd8440ae8f0 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: dump: 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.911 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: _finish_auth 0 2023-09-30T01:06:21.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:21.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 5 monclient: authenticate success, global_id 4713 2023-09-30T01:06:21.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 10 monclient: _renew_subs 2023-09-30T01:06:21.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840010670 con 0x7fd8440ae8f0 2023-09-30T01:06:21.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8440aa3b0 con 0x7fd8440ae8f0 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840010a30 con 0x7fd8440ae8f0 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd8397fa700 10 monclient: dump: 2023-09-30T01:06:21.914 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:21.915 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd83b7fe700 4 set_mon_vals no callback set 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 10 monclient: _renew_subs 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:21.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8440a9140 con 0x7fd8440ae8f0 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.884+0000 7fd84b9a5700 1 librados: init done 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd84b9a5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd84b9a5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd84b9a5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8440a9cc0 con 0x7fd8440ae8f0 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd84001b400 con 0x7fd8440ae8f0 2023-09-30T01:06:21.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.885+0000 7fd8397fa700 1 --2- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 0x7fd82806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd840083a70 con 0x7fd8440ae8f0 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 10 client.4713.objecter ms_dispatch 0x7fd844001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 3 client.4713.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 3 client.4713.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd848f40700 1 --2- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 0x7fd82806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:21.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd848f40700 10 monclient: get_auth_request con 0x7fd828068890 auth_method 0 2023-09-30T01:06:21.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 20 client.4713.objecter dump_active .. 0 homeless 2023-09-30T01:06:21.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd8397fa700 10 client.4713.objecter ms_handle_connect 0x7fd828068890 2023-09-30T01:06:21.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.886+0000 7fd848f40700 1 --2- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 0x7fd82806ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd8340093f0 tx=0x7fd834016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:21.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.890+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd840051330 con 0x7fd8440ae8f0 2023-09-30T01:06:21.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.890+0000 7fd8397fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:21.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:21.890+0000 7fd8397fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:22.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd84b9a5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:22.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd84b9a5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd84b9a5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd844072960 con 0x7fd8440ae8f0 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd8397fa700 1 -- 172.21.15.115:0/3781348357 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd840054950 con 0x7fd8440ae8f0 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd8397fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.112+0000 7fd8397fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:22.113 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 10 librados: watch_flush enter 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 10 librados: watch_flush exit 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 20 client.4713.objecter shutdown clearing up homeless session... 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 10 client.4713.objecter successfully canceled tick 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 msgr2=0x7fd82806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 0x7fd82806ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd8340093f0 tx=0x7fd834016000 comp rx=0 tx=0).stop 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 10 monclient: shutdown 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 msgr2=0x7fd84413e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:22.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd84000c580 tx=0x7fd840007e10 comp rx=0 tx=0).stop 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 shutdown_connections 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> v2:172.21.15.115:6816/115635 conn(0x7fd828068890 0x7fd82806ad40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd8441415c0 0x7fd8440a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd8440ae8f0 0x7fd84413e080 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 --2- 172.21.15.115:0/3781348357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8440aecc0 0x7fd844150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 >> 172.21.15.115:0/3781348357 conn(0x7fd84406a460 msgr2=0x7fd844144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 shutdown_connections 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 -- 172.21.15.115:0/3781348357 wait complete. 2023-09-30T01:06:22.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:22.114+0000 7fd84b9a5700 1 librados: shutdown 2023-09-30T01:06:25.127 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:25.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.255+0000 7f1538503700 10 monclient: get_monmap_and_config 2023-09-30T01:06:25.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.255+0000 7f1538503700 10 monclient: build_initial_monmap 2023-09-30T01:06:25.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.255+0000 7f1538503700 10 monclient: monmap: 2023-09-30T01:06:25.256 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:25.256 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:25.256414+0000 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:25.256414+0000 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:25.257 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.255+0000 7f1538503700 1 Processor -- start 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 -- start start 2023-09-30T01:06:25.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient: init 2023-09-30T01:06:25.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:25.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f15300ae8f0 0x7f15301415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): picked mon.noname-c con 0x7f15300ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15300aecc0 0x7f15300b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): picked mon.noname-a con 0x7f15300aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15300a5c80 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300a7040 con 0x7f15300a5c80 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300a5480 con 0x7f15300aecc0 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300a5760 con 0x7f15300ae8f0 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:25.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1538503700 10 monclient(hunting): authenticate will time out at 1246.706834s 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.256+0000 7f1535a9e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15300aecc0 0x7f15300b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15300aecc0 0x7f15300b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59882/0 (socket says 172.21.15.115:59882) 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 -- 172.21.15.115:0/2003698744 learned_addr learned my addr 172.21.15.115:0/2003698744 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): get_auth_request con 0x7f15300aecc0 auth_method 0 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 1 --2- 172.21.15.115:0/2003698744 >> v2:172.21.15.157:3300/0 conn(0x7f15300ae8f0 0x7f15301415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): get_auth_request con 0x7f15300ae8f0 auth_method 0 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:25.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient(hunting): handle_auth_done global_id 4716 payload 274 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 -- 172.21.15.115:0/2003698744 >> v2:172.21.15.157:3300/0 conn(0x7f15300ae8f0 msgr2=0x7f15301415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 --2- 172.21.15.115:0/2003698744 >> v2:172.21.15.157:3300/0 conn(0x7f15300ae8f0 0x7f15301415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 -- 172.21.15.115:0/2003698744 >> v1:172.21.15.157:6789/0 conn(0x7f15300a5c80 legacy=0x7f15300adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient: _finish_hunting 0 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 monclient: found mon.noname-a 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 -- 172.21.15.115:0/2003698744 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530141b00 con 0x7f15300aecc0 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1535a9e700 1 --2- 172.21.15.115:0/2003698744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f15300b1430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15300b1d50 tx=0x7f152c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540a11edf2175599 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 1 -- 172.21.15.115:0/2003698744 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c009e70 con 0x7f15300aecc0 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: dump: 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:25.275 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: _finish_auth 0 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1538503700 5 monclient: authenticate success, global_id 4716 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1538503700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:25.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 1 -- 172.21.15.115:0/2003698744 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c00b860 con 0x7f15300aecc0 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1538503700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f1538503700 4 set_mon_vals no callback set 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 1 -- 172.21.15.115:0/2003698744 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c00bc00 con 0x7f15300aecc0 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.257+0000 7f153529d700 10 monclient: dump: 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:25.277 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 10 monclient: shutdown 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 -- 172.21.15.115:0/2003698744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 msgr2=0x7f15300b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 --2- 172.21.15.115:0/2003698744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f15300b1430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15300b1d50 tx=0x7f152c009250 comp rx=0 tx=0).stop 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 -- 172.21.15.115:0/2003698744 shutdown_connections 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 --2- 172.21.15.115:0/2003698744 >> v2:172.21.15.157:3300/0 conn(0x7f15300ae8f0 0x7f15301415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 --2- 172.21.15.115:0/2003698744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f15300b1430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 -- 172.21.15.115:0/2003698744 >> 172.21.15.115:0/2003698744 conn(0x7f153006a490 msgr2=0x7f15300630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 -- 172.21.15.115:0/2003698744 shutdown_connections 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 1 -- 172.21.15.115:0/2003698744 wait complete. 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 10 monclient: build_initial_monmap 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.258+0000 7f1538503700 10 monclient: monmap: 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.279 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 librados: starting msgr at 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 librados: starting objecter 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 Processor -- start 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 -- start start 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 librados: setting wanted keys 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 librados: calling monclient init 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient: init 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:25.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 0x7f15300a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): picked mon.noname-c con 0x7f15300a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15300ae8f0 0x7f15300a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): picked mon.noname-b con 0x7f15300ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): picked mon.noname-a con 0x7f15300aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300a9cc0 con 0x7f15300aecc0 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153013d790 con 0x7f15300ae8f0 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:25.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300a7c70 con 0x7f15300a5c80 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1538503700 10 monclient(hunting): authenticate will time out at 1246.709730s 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 0x7f15300a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:59896/0 (socket says 172.21.15.115:59896) 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 1 -- 172.21.15.115:0/3715672542 learned_addr learned my addr 172.21.15.115:0/3715672542 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): get_auth_request con 0x7f15300aecc0 auth_method 0 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:25.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): get_auth_request con 0x7f15300a5c80 auth_method 0 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1535a9e700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15300ae8f0 0x7f15300a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1535a9e700 10 monclient(hunting): get_auth_request con 0x7f15300ae8f0 auth_method 0 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1535a9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1535a9e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1535a9e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f1536aa0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:25.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.259+0000 7f153629f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1535a9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 -- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 msgr2=0x7f15300a9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 0x7f15300a9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 -- 172.21.15.115:0/3715672542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15300ae8f0 msgr2=0x7f15300a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15300ae8f0 0x7f15300a9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 10 monclient: _finish_hunting 0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 monclient: found mon.noname-a 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 -- 172.21.15.115:0/3715672542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c0094f0 con 0x7f15300aecc0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f153629f700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 0x7f15300a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1536aa0700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f152800c8e0 tx=0x7f1528001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 client.?.objecter ms_handle_connect 0x7f15300aecc0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528012630 con 0x7f15300aecc0 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:25.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: dump: 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:25.286 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: _finish_auth 0 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 5 monclient: authenticate success, global_id 4719 2023-09-30T01:06:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1528012790 con 0x7f15300aecc0 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528012b50 con 0x7f15300aecc0 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f15267fc700 10 monclient: dump: 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:25.288 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1534a9c700 4 set_mon_vals no callback set 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 10 monclient: _renew_subs 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153014c070 con 0x7f15300aecc0 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 10 client.4719.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 10 monclient: _renew_subs 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153014f020 con 0x7f15300aecc0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.260+0000 7f1538503700 1 librados: init done 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.261+0000 7f1538503700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.261+0000 7f1538503700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.261+0000 7f1538503700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.261+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153014e640 con 0x7f15300aecc0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.261+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f152801a3a0 con 0x7f15300aecc0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 1 --2- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 0x7f151406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1528084050 con 0x7f15300aecc0 2023-09-30T01:06:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 10 client.4719.objecter ms_dispatch 0x7f1530001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 3 client.4719.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 3 client.4719.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f153629f700 1 --2- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 0x7f151406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f153629f700 10 monclient: get_auth_request con 0x7f15140688c0 auth_method 0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 20 client.4719.objecter dump_active .. 0 homeless 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f15267fc700 10 client.4719.objecter ms_handle_connect 0x7f15140688c0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.262+0000 7f153629f700 1 --2- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 0x7f151406ad70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1520011480 tx=0x7f15200091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.266+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f153014e640 con 0x7f15300aecc0 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.266+0000 7f15267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.266+0000 7f15267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.494+0000 7f1538503700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.494+0000 7f1538503700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.494+0000 7f1538503700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.494+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15300a9e00 con 0x7f15300aecc0 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.495+0000 7f15267fc700 1 -- 172.21.15.115:0/3715672542 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f15300a9e00 con 0x7f15300aecc0 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.495+0000 7f15267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:25.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.495+0000 7f15267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:25.496 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:25.496 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 10 librados: watch_flush enter 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 10 librados: watch_flush exit 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 20 client.4719.objecter shutdown clearing up homeless session... 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 10 client.4719.objecter successfully canceled tick 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 msgr2=0x7f151406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 0x7f151406ad70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1520011480 tx=0x7f15200091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 10 monclient: shutdown 2023-09-30T01:06:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 msgr2=0x7f153014e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f152800c8e0 tx=0x7f1528001710 comp rx=0 tx=0).stop 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 shutdown_connections 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> v2:172.21.15.115:6816/115635 conn(0x7f15140688c0 0x7f151406ad70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15300aecc0 0x7f153014e920 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15300a5c80 0x7f15300a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 --2- 172.21.15.115:0/3715672542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15300ae8f0 0x7f15300a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 >> 172.21.15.115:0/3715672542 conn(0x7f153006a490 msgr2=0x7f153013acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 shutdown_connections 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 -- 172.21.15.115:0/3715672542 wait complete. 2023-09-30T01:06:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:25.497+0000 7f1538503700 1 librados: shutdown 2023-09-30T01:06:28.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.639+0000 7f28dfd81700 10 monclient: get_monmap_and_config 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.639+0000 7f28dfd81700 10 monclient: build_initial_monmap 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.639+0000 7f28dfd81700 10 monclient: monmap: 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:28.640887+0000 2023-09-30T01:06:28.640 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:28.640887+0000 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.640+0000 7f28dfd81700 1 Processor -- start 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.640+0000 7f28dfd81700 1 -- start start 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.640+0000 7f28dfd81700 10 monclient: init 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28d80ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:28.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f28d80aecc0 0x7f28d80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28d81415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80a7040 con 0x7f28d80ae8f0 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80a57f0 con 0x7f28d81415c0 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d80a5480 con 0x7f28d80aecc0 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dfd81700 10 monclient(hunting): authenticate will time out at 1250.091362s 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28ddb1d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f28d80ae8f0 0x7f28d80b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:60516/0 (socket says 172.21.15.115:60516) 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28ddb1d700 1 -- 172.21.15.115:0/656451511 learned_addr learned my addr 172.21.15.115:0/656451511 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:28.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242608238 0 0) 0x7f28d80a7040 con 0x7f28d80ae8f0 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 1 --2- 172.21.15.115:0/656451511 >> v2:172.21.15.157:3300/0 conn(0x7f28d80aecc0 0x7f28d80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): my global_id is 4722 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c4003570 con 0x7f28d80ae8f0 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): get_auth_request con 0x7f28d80aecc0 auth_method 0 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071535092 0 0) 0x7f28d80a57f0 con 0x7f28d81415c0 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 10 monclient(hunting): my global_id is 4633 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d80a7040 con 0x7f28d81415c0 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.641+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275875053 0 0) 0x7f28c4003570 con 0x7f28d80ae8f0 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 >> v2:172.21.15.157:3300/0 conn(0x7f28d80aecc0 msgr2=0x7f28d80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 --2- 172.21.15.115:0/656451511 >> v2:172.21.15.157:3300/0 conn(0x7f28d80aecc0 0x7f28d80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 >> v1:172.21.15.157:6789/0 conn(0x7f28d81415c0 legacy=0x7f28d80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: _finish_hunting 0 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:06:28.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80a55c0 con 0x7f28d80ae8f0 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: _finish_auth 0 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f28c80013f0 con 0x7f28d80ae8f0 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 5 monclient: authenticate success, global_id 4722 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:06:28.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: dump: 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:28.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 >> v1:172.21.15.115:6789/0 conn(0x7f28d80ae8f0 legacy=0x7f28d80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28c4004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): picked mon.a con 0x7f28c4003c40 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28c4004810 0x7f28c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): picked mon.b con 0x7f28c4004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28c4005210 0x7f28c400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): picked mon.c con 0x7f28c4005210 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80a5480 con 0x7f28c4003c40 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c400c130 con 0x7f28c4004810 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c400c2c0 con 0x7f28c4005210 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28c4004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): get_auth_request con 0x7f28c4003c40 auth_method 0 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28c4004810 0x7f28c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28c4005210 0x7f28c400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): get_auth_request con 0x7f28c4004810 auth_method 0 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): get_auth_request con 0x7f28c4005210 auth_method 0 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:28.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dd31c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_done global_id 4722 payload 293 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 -- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28c4005210 msgr2=0x7f28c400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28c4005210 0x7f28c400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 -- 172.21.15.115:0/656451511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28c4004810 msgr2=0x7f28c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28c4004810 0x7f28c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient: _finish_hunting 0 2023-09-30T01:06:28.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 monclient: found mon.a 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 -- 172.21.15.115:0/656451511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c4003570 con 0x7f28c4003c40 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 10 monclient: _finish_auth 0 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28de31e700 1 -- 172.21.15.115:0/656451511 reap_dead start 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28c4004030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c190 tx=0x7f28c8001cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa7c6db8f443821 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800ca40 con 0x7f28c4003c40 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:28.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: dump: 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:28.669 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c800cbe0 con 0x7f28c4003c40 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 1 -- 172.21.15.115:0/656451511 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8005920 con 0x7f28c4003c40 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dfd81700 4 set_mon_vals no callback set 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:28.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.642+0000 7f28dcb1b700 10 monclient: dump: 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:28.671 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 10 monclient: shutdown 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 -- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 msgr2=0x7f28c4004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28c4004030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c190 tx=0x7f28c8001cc0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 -- 172.21.15.115:0/656451511 shutdown_connections 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 --2- 172.21.15.115:0/656451511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28c4004030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 -- 172.21.15.115:0/656451511 >> 172.21.15.115:0/656451511 conn(0x7f28d806a460 msgr2=0x7f28d80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 -- 172.21.15.115:0/656451511 shutdown_connections 2023-09-30T01:06:28.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.643+0000 7f28dfd81700 1 -- 172.21.15.115:0/656451511 wait complete. 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 10 monclient: build_initial_monmap 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 10 monclient: monmap: 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.673 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 librados: starting msgr at 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 librados: starting objecter 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 Processor -- start 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 -- start start 2023-09-30T01:06:28.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 librados: setting wanted keys 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 1 librados: calling monclient init 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.644+0000 7f28dfd81700 10 monclient: init 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28d813d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-a con 0x7f28c4003c40 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28d80a92d0 0x7f28d814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-b con 0x7f28d80a92d0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:28.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): picked mon.noname-c con 0x7f28d814e8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d814df20 con 0x7f28c4003c40 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d814dc40 con 0x7f28d80a92d0 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80a7950 con 0x7f28d814e8f0 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient(hunting): authenticate will time out at 1250.095099s 2023-09-30T01:06:28.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28d813d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48230/0 (socket says 172.21.15.115:48230) 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): get_auth_request con 0x7f28c4003c40 auth_method 0 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 -- 172.21.15.115:0/3547003501 learned_addr learned my addr 172.21.15.115:0/3547003501 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): get_auth_request con 0x7f28d814e8f0 auth_method 0 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dd31c700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28d80a92d0 0x7f28d814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dd31c700 10 monclient(hunting): get_auth_request con 0x7f28d80a92d0 auth_method 0 2023-09-30T01:06:28.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dd31c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dd31c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dd31c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient(hunting): handle_auth_done global_id 4541 payload 824 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 -- 172.21.15.115:0/3547003501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28d80a92d0 msgr2=0x7f28d814e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28d80a92d0 0x7f28d814e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 -- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 msgr2=0x7f28d813d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28d813d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient: _finish_hunting 0 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 monclient: found mon.noname-c 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:28.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 -- 172.21.15.115:0/3547003501 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80a8590 con 0x7f28d814e8f0 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28d813d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28de31e700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d80b0340 tx=0x7f28d00017a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 client.?.objecter ms_handle_connect 0x7f28d814e8f0 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0006db0 con 0x7f28d814e8f0 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:28.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: dump: 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:28.681 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: _finish_auth 0 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 5 monclient: authenticate success, global_id 4541 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d000c670 con 0x7f28d814e8f0 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient: _renew_subs 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d80a9740 con 0x7f28d814e8f0 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:28.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d000ca30 con 0x7f28d814e8f0 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cdffb700 10 monclient: dump: 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:28.683 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.645+0000 7f28cffff700 4 set_mon_vals no callback set 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 monclient: _renew_subs 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d81518d0 con 0x7f28d814e8f0 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 1 librados: init done 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:28.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.646+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d80aad40 con 0x7f28d814e8f0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f28d001d460 con 0x7f28d814e8f0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 1 --2- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 0x7f28bc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 0x7f28bc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f28d0083e90 con 0x7f28d814e8f0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 10 client.4541.objecter ms_dispatch 0x7f28d8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 3 client.4541.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 3 client.4541.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28ddb1d700 10 monclient: get_auth_request con 0x7f28bc068890 auth_method 0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.647+0000 7f28cdffb700 20 client.4541.objecter dump_active .. 0 homeless 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.648+0000 7f28cdffb700 10 client.4541.objecter ms_handle_connect 0x7f28bc068890 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.648+0000 7f28ddb1d700 1 --2- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 0x7f28bc06ad40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c190 tx=0x7f28c8015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:28.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.652+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28d0051750 con 0x7f28d814e8f0 2023-09-30T01:06:28.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.652+0000 7f28cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:28.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.652+0000 7f28cdffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:28.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28dfd81700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:28.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28dfd81700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:28.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28dfd81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:28.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28d814e0d0 con 0x7f28d814e8f0 2023-09-30T01:06:28.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28cdffb700 1 -- 172.21.15.115:0/3547003501 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f28d814e0d0 con 0x7f28d814e8f0 2023-09-30T01:06:28.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:28.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.876+0000 7f28cdffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:28.877 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:28.878 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:28.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 10 librados: watch_flush enter 2023-09-30T01:06:28.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 10 librados: watch_flush exit 2023-09-30T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 20 client.4541.objecter shutdown clearing up homeless session... 2023-09-30T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 10 client.4541.objecter successfully canceled tick 2023-09-30T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 msgr2=0x7f28bc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 0x7f28bc06ad40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c190 tx=0x7f28c8015000 comp rx=0 tx=0).stop 2023-09-30T01:06:28.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 10 monclient: shutdown 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 msgr2=0x7f28d81511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28d80b0340 tx=0x7f28d00017a0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 shutdown_connections 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> v2:172.21.15.115:6816/115635 conn(0x7f28bc068890 0x7f28bc06ad40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28d814e8f0 0x7f28d81511d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28c4003c40 0x7f28d813d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 --2- 172.21.15.115:0/3547003501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28d80a92d0 0x7f28d814e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 >> 172.21.15.115:0/3547003501 conn(0x7f28d806a460 msgr2=0x7f28d8140a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 shutdown_connections 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 -- 172.21.15.115:0/3547003501 wait complete. 2023-09-30T01:06:28.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:28.878+0000 7f28dfd81700 1 librados: shutdown 2023-09-30T01:06:31.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 10 monclient: get_monmap_and_config 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 10 monclient: build_initial_monmap 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 10 monclient: monmap: 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:32.019998+0000 2023-09-30T01:06:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:32.019998+0000 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:32.020 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 1 Processor -- start 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 1 -- start start 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.019+0000 7f09c25b3700 10 monclient: init 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:06:32.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09bc0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-b con 0x7f09bc0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09bc1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc0a7040 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc0a57f0 con 0x7f09bc1415c0 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0a5480 con 0x7f09bc0aecc0 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09c25b3700 10 monclient(hunting): authenticate will time out at 1253.470463s 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bbfff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f09bc0ae8f0 0x7f09bc0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:58070/0 (socket says 172.21.15.115:58070) 2023-09-30T01:06:32.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bbfff700 1 -- 172.21.15.115:0/2075088344 learned_addr learned my addr 172.21.15.115:0/2075088344 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request con 0x7f09bc0aecc0 auth_method 0 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 msgr2=0x7f09bc0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 msgr2=0x7f09bc0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429928319 0 0) 0x7f09bc0a7040 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09baffd700 10 monclient(hunting): my global_id is 4728 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.020+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a4003570 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706763445 0 0) 0x7f09bc0a57f0 con 0x7f09bc1415c0 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): my global_id is 4639 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09bc0a7040 con 0x7f09bc1415c0 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383747651 0 0) 0x7f09a4003570 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 msgr2=0x7f09bc0a5d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 >> v1:172.21.15.157:6789/0 conn(0x7f09bc1415c0 legacy=0x7f09bc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: _finish_hunting 0 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0a55c0 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: _finish_auth 0 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f09b0001470 con 0x7f09bc0ae8f0 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 5 monclient: authenticate success, global_id 4728 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:32.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: dump: 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:32.036 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 >> v1:172.21.15.115:6789/0 conn(0x7f09bc0ae8f0 legacy=0x7f09bc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09a4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): picked mon.c con 0x7f09bc0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09a40041d0 0x7f09a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): picked mon.a con 0x7f09a40041d0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a4004dc0 0x7f09a4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): picked mon.b con 0x7f09a4004dc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0a5480 con 0x7f09a40041d0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a400bcc0 con 0x7f09a4004dc0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a400be80 con 0x7f09bc0aecc0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09a4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09a40041d0 0x7f09a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): get_auth_request con 0x7f09bc0aecc0 auth_method 0 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request con 0x7f09a40041d0 auth_method 0 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a4004dc0 0x7f09a4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:32.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): get_auth_request con 0x7f09a4004dc0 auth_method 0 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient(hunting): handle_auth_done global_id 4728 payload 293 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a4004dc0 msgr2=0x7f09a4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a4004dc0 0x7f09a4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09a40041d0 msgr2=0x7f09a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09a40041d0 0x7f09a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 monclient: found mon.c 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 -- 172.21.15.115:0/2075088344 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4003570 con 0x7f09bc0aecc0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 10 monclient: _finish_auth 0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09a40041d0 0x7f09a4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09bbfff700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09a4003c20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c1e0 tx=0x7f09b0001d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b1faa82eb81c18 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b000cb10 con 0x7f09bc0aecc0 2023-09-30T01:06:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: dump: 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:32.042 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b000cc90 con 0x7f09bc0aecc0 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 1 -- 172.21.15.115:0/2075088344 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0005b70 con 0x7f09bc0aecc0 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09c25b3700 4 set_mon_vals no callback set 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:32.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.021+0000 7f09baffd700 10 monclient: dump: 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:32.045 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 10 monclient: shutdown 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 -- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 msgr2=0x7f09a4003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 --2- 172.21.15.115:0/2075088344 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09a4003c20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c1e0 tx=0x7f09b0001d40 comp rx=0 tx=0).stop 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c0b50700 1 -- 172.21.15.115:0/2075088344 reap_dead start 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 -- 172.21.15.115:0/2075088344 shutdown_connections 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 -- 172.21.15.115:0/2075088344 >> 172.21.15.115:0/2075088344 conn(0x7f09bc06a460 msgr2=0x7f09bc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 -- 172.21.15.115:0/2075088344 shutdown_connections 2023-09-30T01:06:32.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.022+0000 7f09c25b3700 1 -- 172.21.15.115:0/2075088344 wait complete. 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 10 monclient: build_initial_monmap 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 10 monclient: monmap: 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 librados: starting msgr at 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 librados: starting objecter 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 Processor -- start 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 -- start start 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 librados: setting wanted keys 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 1 librados: calling monclient init 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.023+0000 7f09c25b3700 10 monclient: init 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-c con 0x7f09bc0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09bc0a95c0 0x7f09bc13e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-b con 0x7f09bc0a95c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 0x7f09bc0a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): picked mon.noname-a con 0x7f09bc151070 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0ab480 con 0x7f09bc151070 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0ab760 con 0x7f09bc0a95c0 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0a9dd0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient(hunting): authenticate will time out at 1253.474265s 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:37792/0 (socket says 172.21.15.115:37792) 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 -- 172.21.15.115:0/515540602 learned_addr learned my addr 172.21.15.115:0/515540602 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): get_auth_request con 0x7f09bc0aecc0 auth_method 0 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 0x7f09bc0a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): get_auth_request con 0x7f09bc151070 auth_method 0 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:32.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09bc0a95c0 0x7f09bc13e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request con 0x7f09bc0a95c0 auth_method 0 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 -- 172.21.15.115:0/515540602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09bc0a95c0 msgr2=0x7f09bc13e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09bc0a95c0 0x7f09bc13e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 -- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 msgr2=0x7f09bc0a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 0x7f09bc0a7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 monclient: found mon.noname-c 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 -- 172.21.15.115:0/515540602 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc13eeb0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c0b50700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 0x7f09bc0a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09bbfff700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c7d0 tx=0x7f09b0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 client.?.objecter ms_handle_connect 0x7f09bc0aecc0 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:32.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0001650 con 0x7f09bc0aecc0 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 monclient: dump: 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:32.053 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 10 monclient: _finish_auth 0 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 5 monclient: authenticate success, global_id 4544 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00070d0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient: _renew_subs 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.024+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc1514c0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09a3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0007490 con 0x7f09bc0aecc0 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09a3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:32.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09a3fff700 10 monclient: dump: 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:32.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 monclient: _renew_subs 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09ba7fc700 4 set_mon_vals no callback set 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc0a7ca0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 1 librados: init done 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.025+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc0aa140 con 0x7f09bc0aecc0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f09b0028450 con 0x7f09bc0aecc0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 1 --2- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 0x7f09a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f09b0085870 con 0x7f09bc0aecc0 2023-09-30T01:06:32.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 10 client.4544.objecter ms_dispatch 0x7f09bc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 3 client.4544.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 3 client.4544.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 0x7f09a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09bb7fe700 10 monclient: get_auth_request con 0x7f09a8068890 auth_method 0 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.026+0000 7f09a3fff700 20 client.4544.objecter dump_active .. 0 homeless 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.027+0000 7f09a3fff700 10 client.4544.objecter ms_handle_connect 0x7f09a8068890 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.027+0000 7f09bb7fe700 1 --2- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 0x7f09a806ad40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09bc13e010 tx=0x7f09ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.031+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09b00531f0 con 0x7f09bc0aecc0 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.031+0000 7f09a3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:32.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.031+0000 7f09a3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:32.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.251+0000 7f09c25b3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.251+0000 7f09c25b3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.251+0000 7f09c25b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.251+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09bc072960 con 0x7f09bc0aecc0 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.252+0000 7f09a3fff700 1 -- 172.21.15.115:0/515540602 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f09b0056810 con 0x7f09bc0aecc0 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.252+0000 7f09a3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.252+0000 7f09a3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:32.254 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:32.255 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.253+0000 7f09c25b3700 10 librados: watch_flush enter 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.253+0000 7f09c25b3700 10 librados: watch_flush exit 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.253+0000 7f09c25b3700 20 client.4544.objecter shutdown clearing up homeless session... 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.253+0000 7f09c25b3700 10 client.4544.objecter successfully canceled tick 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 msgr2=0x7f09a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 0x7f09a806ad40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09bc13e010 tx=0x7f09ac008040 comp rx=0 tx=0).stop 2023-09-30T01:06:32.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 10 monclient: shutdown 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 msgr2=0x7f09bc0aa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c7d0 tx=0x7f09b0015000 comp rx=0 tx=0).stop 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 shutdown_connections 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> v2:172.21.15.115:6816/115635 conn(0x7f09a8068890 0x7f09a806ad40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09bc151070 0x7f09bc0a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f09bc0aecc0 0x7f09bc0aa2f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 --2- 172.21.15.115:0/515540602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09bc0a95c0 0x7f09bc13e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:32.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 >> 172.21.15.115:0/515540602 conn(0x7f09bc06a460 msgr2=0x7f09bc144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:32.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 shutdown_connections 2023-09-30T01:06:32.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 -- 172.21.15.115:0/515540602 wait complete. 2023-09-30T01:06:32.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:32.254+0000 7f09c25b3700 1 librados: shutdown 2023-09-30T01:06:35.266 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:35.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.396+0000 7f51cb308700 10 monclient: get_monmap_and_config 2023-09-30T01:06:35.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.396+0000 7f51cb308700 10 monclient: build_initial_monmap 2023-09-30T01:06:35.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.396+0000 7f51cb308700 10 monclient: monmap: 2023-09-30T01:06:35.396 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:35.397322+0000 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:35.397322+0000 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:35.397 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.396+0000 7f51cb308700 1 Processor -- start 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 -- start start 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient: init 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f51c40ae8f0 0x7f51c40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-a con 0x7f51c40ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-b con 0x7f51c40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51c41415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40a7040 con 0x7f51c41415c0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c40b19b0 con 0x7f51c40ae8f0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40a57f0 con 0x7f51c40aecc0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51cb308700 10 monclient(hunting): authenticate will time out at 1256.847762s 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c90a4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f51c40ae8f0 0x7f51c40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c88a3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c98a5700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f51c41415c0 0x7f51c40adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:58074/0 (socket says 172.21.15.115:58074) 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c98a5700 1 -- 172.21.15.115:0/371296452 learned_addr learned my addr 172.21.15.115:0/371296452 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c90a4700 10 monclient(hunting): get_auth_request con 0x7f51c40ae8f0 auth_method 0 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c90a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c90a4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c90a4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c88a3700 10 monclient(hunting): get_auth_request con 0x7f51c40aecc0 auth_method 0 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c88a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c88a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.397+0000 7f51c88a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:35.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 -- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 msgr2=0x7f51c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c40b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 -- 172.21.15.115:0/371296452 >> v1:172.21.15.115:6789/0 conn(0x7f51c41415c0 legacy=0x7f51c40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient: _finish_hunting 0 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 monclient: found mon.noname-a 2023-09-30T01:06:35.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 -- 172.21.15.115:0/371296452 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40a5480 con 0x7f51c40ae8f0 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 1 -- 172.21.15.115:0/371296452 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690057348 0 0) 0x7f51c40a7040 con 0x7f51c41415c0 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c88a3700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51c90a4700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c40ae8f0 0x7f51c40a5d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c7732ccf99672e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 1 -- 172.21.15.115:0/371296452 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400b640 con 0x7f51c40ae8f0 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: dump: 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:35.411 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: _finish_auth 0 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 5 monclient: authenticate success, global_id 4734 2023-09-30T01:06:35.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 1 -- 172.21.15.115:0/371296452 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b400b7a0 con 0x7f51c40ae8f0 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 4 set_mon_vals no callback set 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 1 -- 172.21.15.115:0/371296452 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400bb40 con 0x7f51c40ae8f0 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51bbfff700 10 monclient: dump: 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:35.413 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 10 monclient: shutdown 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 1 -- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c40ae8f0 msgr2=0x7f51c40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.398+0000 7f51cb308700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c40ae8f0 0x7f51c40a5d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 -- 172.21.15.115:0/371296452 shutdown_connections 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c40ae8f0 0x7f51c40a5d10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 --2- 172.21.15.115:0/371296452 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 -- 172.21.15.115:0/371296452 >> 172.21.15.115:0/371296452 conn(0x7f51c406a460 msgr2=0x7f51c40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 -- 172.21.15.115:0/371296452 shutdown_connections 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 -- 172.21.15.115:0/371296452 wait complete. 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 10 monclient: build_initial_monmap 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 10 monclient: monmap: 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 librados: starting msgr at 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.399+0000 7f51cb308700 1 librados: starting objecter 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 Processor -- start 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 -- start start 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 librados: setting wanted keys 2023-09-30T01:06:35.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 librados: calling monclient init 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient: init 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c40ae8f0 0x7f51c40aa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-b con 0x7f51c40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-c con 0x7f51c40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 0x7f51c4151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): picked mon.noname-a con 0x7f51c41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c414e1a0 con 0x7f51c41415c0 2023-09-30T01:06:35.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c413eeb0 con 0x7f51c40ae8f0 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c413f020 con 0x7f51c40aecc0 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51cb308700 10 monclient(hunting): authenticate will time out at 1256.850800s 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c98a5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 0x7f51c4151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:37804/0 (socket says 172.21.15.115:37804) 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 1 -- 172.21.15.115:0/425362745 learned_addr learned my addr 172.21.15.115:0/425362745 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): get_auth_request con 0x7f51c40aecc0 auth_method 0 2023-09-30T01:06:35.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c98a5700 10 monclient(hunting): get_auth_request con 0x7f51c41415c0 auth_method 0 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c98a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c98a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c98a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.400+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c98a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:35.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c98a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c98a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 10 monclient(hunting): handle_auth_done global_id 4550 payload 824 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c90a4700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c40ae8f0 0x7f51c40aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 -- 172.21.15.115:0/425362745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c40ae8f0 msgr2=0x7f51c40aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c40ae8f0 0x7f51c40aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 -- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 msgr2=0x7f51c4151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 0x7f51c4151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 10 monclient: _finish_hunting 0 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 monclient: found mon.noname-c 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 -- 172.21.15.115:0/425362745 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0009420 con 0x7f51c40aecc0 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c98a5700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 0x7f51c4151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51c88a3700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51c000ec80 tx=0x7f51c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 client.?.objecter ms_handle_connect 0x7f51c40aecc0 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c000c8f0 con 0x7f51c40aecc0 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: dump: 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: _finish_auth 0 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 5 monclient: authenticate success, global_id 4550 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c000ca50 con 0x7f51c40aecc0 2023-09-30T01:06:35.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c000cdd0 con 0x7f51c40aecc0 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51b97fa700 10 monclient: dump: 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51bb7fe700 4 set_mon_vals no callback set 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 10 monclient: _renew_subs 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b4009420 con 0x7f51c40aecc0 2023-09-30T01:06:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 10 monclient: _renew_subs 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c40aad20 con 0x7f51c40aecc0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.401+0000 7f51cb308700 1 librados: init done 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.402+0000 7f51cb308700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.402+0000 7f51cb308700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.402+0000 7f51cb308700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.402+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c40b0660 con 0x7f51c40aecc0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.402+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f51c0013680 con 0x7f51c40aecc0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 1 --2- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 0x7f51a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f51c0084250 con 0x7f51c40aecc0 2023-09-30T01:06:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 10 client.4550.objecter ms_dispatch 0x7f51c4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 3 client.4550.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 3 client.4550.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51c90a4700 1 --2- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 0x7f51a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51c90a4700 10 monclient: get_auth_request con 0x7f51a8068890 auth_method 0 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 20 client.4550.objecter dump_active .. 0 homeless 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51b97fa700 10 client.4550.objecter ms_handle_connect 0x7f51a8068890 2023-09-30T01:06:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.403+0000 7f51c90a4700 1 --2- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 0x7f51a806ad40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f51b4003080 tx=0x7f51b4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:35.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.407+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51c0051b10 con 0x7f51c40aecc0 2023-09-30T01:06:35.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.407+0000 7f51b97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:35.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.407+0000 7f51b97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.633+0000 7f51cb308700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.633+0000 7f51cb308700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.633+0000 7f51cb308700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.633+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f51c414e310 con 0x7f51c40aecc0 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.633+0000 7f51b97fa700 1 -- 172.21.15.115:0/425362745 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f51c0055130 con 0x7f51c40aecc0 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.634+0000 7f51b97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.634+0000 7f51b97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:35.635 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:35.635 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:35.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 10 librados: watch_flush enter 2023-09-30T01:06:35.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 10 librados: watch_flush exit 2023-09-30T01:06:35.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 20 client.4550.objecter shutdown clearing up homeless session... 2023-09-30T01:06:35.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 10 client.4550.objecter successfully canceled tick 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 msgr2=0x7f51a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 0x7f51a806ad40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f51b4003080 tx=0x7f51b4017000 comp rx=0 tx=0).stop 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 10 monclient: shutdown 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 msgr2=0x7f51c414e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.635+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51c000ec80 tx=0x7f51c00091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 shutdown_connections 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> v2:172.21.15.115:6816/115635 conn(0x7f51a8068890 0x7f51a806ad40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51c41415c0 0x7f51c4151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c40ae8f0 0x7f51c40aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 --2- 172.21.15.115:0/425362745 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f51c40aecc0 0x7f51c414e630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 >> 172.21.15.115:0/425362745 conn(0x7f51c406a460 msgr2=0x7f51c40adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 shutdown_connections 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 -- 172.21.15.115:0/425362745 wait complete. 2023-09-30T01:06:35.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:35.636+0000 7f51cb308700 1 librados: shutdown 2023-09-30T01:06:38.648 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.776+0000 7f49f7ccd700 10 monclient: get_monmap_and_config 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.776+0000 7f49f7ccd700 10 monclient: build_initial_monmap 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.776+0000 7f49f7ccd700 10 monclient: monmap: 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:38.777942+0000 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:38.777942+0000 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:38.778 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:38.778 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:38.778 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:38.778 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:38.778 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.777+0000 7f49f7ccd700 1 Processor -- start 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.777+0000 7f49f7ccd700 1 -- start start 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.777+0000 7f49f7ccd700 10 monclient: init 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f49f00ae8f0 0x7f49f00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-a con 0x7f49f00ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49f00aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49f01415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f00a7040 con 0x7f49f00aecc0 2023-09-30T01:06:38.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f00a57f0 con 0x7f49f01415c0 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f00a5480 con 0x7f49f00ae8f0 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f7ccd700 10 monclient(hunting): authenticate will time out at 1260.228434s 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f49f00ae8f0 0x7f49f00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5268700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f49f00aecc0 0x7f49f00b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:58076/0 (socket says 172.21.15.115:58076) 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f49f00ae8f0 0x7f49f00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47388/0 (socket says 172.21.15.115:47388) 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5268700 1 -- 172.21.15.115:0/1177733569 learned_addr learned my addr 172.21.15.115:0/1177733569 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): get_auth_request con 0x7f49f00ae8f0 auth_method 0 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:38.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient(hunting): handle_auth_done global_id 4743 payload 274 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 -- 172.21.15.115:0/1177733569 >> v1:172.21.15.157:6789/0 conn(0x7f49f01415c0 legacy=0x7f49f00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 -- 172.21.15.115:0/1177733569 >> v1:172.21.15.115:6789/0 conn(0x7f49f00aecc0 legacy=0x7f49f00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient: _finish_hunting 0 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 monclient: found mon.noname-a 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.778+0000 7f49f5a69700 1 -- 172.21.15.115:0/1177733569 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00a55c0 con 0x7f49f00ae8f0 2023-09-30T01:06:38.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f5a69700 1 --2- 172.21.15.115:0/1177733569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a5d10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0098e0 tx=0x7f49ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4450973efd967d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 1 -- 172.21.15.115:0/1177733569 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec009e80 con 0x7f49f00ae8f0 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: dump: 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:38.789 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: _finish_auth 0 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 5 monclient: authenticate success, global_id 4743 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 1 -- 172.21.15.115:0/1177733569 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec00b820 con 0x7f49f00ae8f0 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 1 -- 172.21.15.115:0/1177733569 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00bba0 con 0x7f49f00ae8f0 2023-09-30T01:06:38.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 4 set_mon_vals no callback set 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f4a67700 10 monclient: dump: 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:38.791 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 10 monclient: shutdown 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 -- 172.21.15.115:0/1177733569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f00ae8f0 msgr2=0x7f49f00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/1177733569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a5d10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0098e0 tx=0x7f49ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 -- 172.21.15.115:0/1177733569 shutdown_connections 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/1177733569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a5d10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 -- 172.21.15.115:0/1177733569 >> 172.21.15.115:0/1177733569 conn(0x7f49f006a460 msgr2=0x7f49f00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 -- 172.21.15.115:0/1177733569 shutdown_connections 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.779+0000 7f49f7ccd700 1 -- 172.21.15.115:0/1177733569 wait complete. 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 10 monclient: build_initial_monmap 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 10 monclient: monmap: 2023-09-30T01:06:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 librados: starting msgr at 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 librados: starting objecter 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 Processor -- start 2023-09-30T01:06:38.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 -- start start 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 librados: setting wanted keys 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 1 librados: calling monclient init 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.780+0000 7f49f7ccd700 10 monclient: init 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-b con 0x7f49f00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49f00aecc0 0x7f49f00a8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-c con 0x7f49f00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): picked mon.noname-a con 0x7f49f01415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:38.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00a99d0 con 0x7f49f01415c0 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00aa760 con 0x7f49f00ae8f0 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f013d380 con 0x7f49f00aecc0 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:38.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f7ccd700 10 monclient(hunting): authenticate will time out at 1260.231414s 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:47398/0 (socket says 172.21.15.115:47398) 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 -- 172.21.15.115:0/613486454 learned_addr learned my addr 172.21.15.115:0/613486454 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): get_auth_request con 0x7f49f01415c0 auth_method 0 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49f00aecc0 0x7f49f00a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:38.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): get_auth_request con 0x7f49f00aecc0 auth_method 0 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5268700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 10 monclient(hunting): get_auth_request con 0x7f49f00ae8f0 auth_method 0 2023-09-30T01:06:38.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 -- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49f00aecc0 msgr2=0x7f49f00a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49f00aecc0 0x7f49f00a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 -- 172.21.15.115:0/613486454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 msgr2=0x7f49f00a9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient: _finish_hunting 0 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 monclient: found mon.noname-a 2023-09-30T01:06:38.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f626a700 1 -- 172.21.15.115:0/613486454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec009420 con 0x7f49f01415c0 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.781+0000 7f49f5a69700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f626a700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c930 tx=0x7f49e80026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 client.?.objecter ms_handle_connect 0x7f49f01415c0 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8012630 con 0x7f49f01415c0 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: dump: 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e80127d0 con 0x7f49f01415c0 2023-09-30T01:06:38.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 5 monclient: authenticate success, global_id 4749 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 10 monclient: _renew_subs 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f00aa9d0 con 0x7f49f01415c0 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8012b90 con 0x7f49f01415c0 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e1ffb700 10 monclient: dump: 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.801 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49e3fff700 4 set_mon_vals no callback set 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 10 monclient: _renew_subs 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f00a7840 con 0x7f49f01415c0 2023-09-30T01:06:38.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.782+0000 7f49f7ccd700 1 librados: init done 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.783+0000 7f49f7ccd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.783+0000 7f49f7ccd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.783+0000 7f49f7ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.783+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f0072960 con 0x7f49f01415c0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.783+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f49e801b400 con 0x7f49f01415c0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 1 --2- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 0x7f49d406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49f5a69700 1 --2- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 0x7f49d406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f49e8084290 con 0x7f49f01415c0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 10 client.4749.objecter ms_dispatch 0x7f49f0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 3 client.4749.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 3 client.4749.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:38.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49f5a69700 10 monclient: get_auth_request con 0x7f49d4068890 auth_method 0 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 20 client.4749.objecter dump_active .. 0 homeless 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49e1ffb700 10 client.4749.objecter ms_handle_connect 0x7f49d4068890 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.784+0000 7f49f5a69700 1 --2- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 0x7f49d406ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0030c0 tx=0x7f49ec016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.788+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49e8051c00 con 0x7f49f01415c0 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.788+0000 7f49e1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:38.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:38.788+0000 7f49e1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49f7ccd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49f7ccd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49f7ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49f0141a40 con 0x7f49f01415c0 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49e1ffb700 1 -- 172.21.15.115:0/613486454 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f49e8055220 con 0x7f49f01415c0 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49e1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:39.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.011+0000 7f49e1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:39.016 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:39.016 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 10 librados: watch_flush enter 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 10 librados: watch_flush exit 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 20 client.4749.objecter shutdown clearing up homeless session... 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 10 client.4749.objecter successfully canceled tick 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 msgr2=0x7f49d406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 0x7f49d406ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0030c0 tx=0x7f49ec016000 comp rx=0 tx=0).stop 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 10 monclient: shutdown 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 msgr2=0x7f49f01514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:39.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c930 tx=0x7f49e80026e0 comp rx=0 tx=0).stop 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 shutdown_connections 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> v2:172.21.15.115:6816/115635 conn(0x7f49d4068890 0x7f49d406ad40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49f01415c0 0x7f49f01514e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49f00ae8f0 0x7f49f00a9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.013+0000 7f49f7ccd700 1 --2- 172.21.15.115:0/613486454 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f49f00aecc0 0x7f49f00a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.014+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 >> 172.21.15.115:0/613486454 conn(0x7f49f006a460 msgr2=0x7f49f00adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.014+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 shutdown_connections 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.014+0000 7f49f7ccd700 1 -- 172.21.15.115:0/613486454 wait complete. 2023-09-30T01:06:39.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:39.014+0000 7f49f7ccd700 1 librados: shutdown 2023-09-30T01:06:42.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:42.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 10 monclient: get_monmap_and_config 2023-09-30T01:06:42.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 10 monclient: build_initial_monmap 2023-09-30T01:06:42.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 10 monclient: monmap: 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:42.156228+0000 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:42.156228+0000 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:42.165 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 1 Processor -- start 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 1 -- start start 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.155+0000 7fe40c4e6700 10 monclient: init 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe4040ae8f0 0x7fe4040a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-a con 0x7fe4040ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4040aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4041415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040a7040 con 0x7fe4040aecc0 2023-09-30T01:06:42.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040a57f0 con 0x7fe4041415c0 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4040a5480 con 0x7fe4040ae8f0 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40c4e6700 10 monclient(hunting): authenticate will time out at 1263.606720s 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe4040ae8f0 0x7fe4040a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe4040ae8f0 0x7fe4040a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40782/0 (socket says 172.21.15.115:40782) 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe409a81700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe4040aecc0 0x7fe4040b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57956/0 (socket says 172.21.15.115:57956) 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 1 -- 172.21.15.115:0/4220351225 learned_addr learned my addr 172.21.15.115:0/4220351225 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): get_auth_request con 0x7fe4040ae8f0 auth_method 0 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:42.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.156+0000 7fe40a282700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 10 monclient(hunting): handle_auth_done global_id 4752 payload 274 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 1 -- 172.21.15.115:0/4220351225 >> v1:172.21.15.157:6789/0 conn(0x7fe4041415c0 legacy=0x7fe4040adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 1 -- 172.21.15.115:0/4220351225 >> v1:172.21.15.115:6789/0 conn(0x7fe4040aecc0 legacy=0x7fe4040b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 10 monclient: _finish_hunting 0 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 1 monclient: found mon.noname-a 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 1 -- 172.21.15.115:0/4220351225 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040a55c0 con 0x7fe4040ae8f0 2023-09-30T01:06:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 1 -- 172.21.15.115:0/4220351225 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338125572 0 0) 0x7fe4040a7040 con 0x7fe4040aecc0 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40a282700 1 --2- 172.21.15.115:0/4220351225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4040ae8f0 0x7fe4040a5d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40099b0 tx=0x7fe3f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48362bce18146437 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 1 -- 172.21.15.115:0/4220351225 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400b820 con 0x7fe4040ae8f0 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: dump: 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.169 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: _finish_auth 0 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 1 -- 172.21.15.115:0/4220351225 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f400b980 con 0x7fe4040ae8f0 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 5 monclient: authenticate success, global_id 4752 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 1 -- 172.21.15.115:0/4220351225 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400bd20 con 0x7fe4040ae8f0 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 4 set_mon_vals no callback set 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe409280700 10 monclient: dump: 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.171 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 10 monclient: shutdown 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 1 -- 172.21.15.115:0/4220351225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4040ae8f0 msgr2=0x7fe4040a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.157+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/4220351225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4040ae8f0 0x7fe4040a5d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40099b0 tx=0x7fe3f4009250 comp rx=0 tx=0).stop 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 -- 172.21.15.115:0/4220351225 shutdown_connections 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/4220351225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4040ae8f0 0x7fe4040a5d10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 -- 172.21.15.115:0/4220351225 >> 172.21.15.115:0/4220351225 conn(0x7fe40406a460 msgr2=0x7fe4040630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 -- 172.21.15.115:0/4220351225 shutdown_connections 2023-09-30T01:06:42.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 -- 172.21.15.115:0/4220351225 wait complete. 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 10 monclient: build_initial_monmap 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 10 monclient: monmap: 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:42.173 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 librados: starting msgr at 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 librados: starting objecter 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.158+0000 7fe40c4e6700 1 Processor -- start 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 -- start start 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 librados: setting wanted keys 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 librados: calling monclient init 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient: init 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 0x7fe4040aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-b con 0x7fe4040ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:42.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-c con 0x7fe4040aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): picked mon.noname-a con 0x7fe4041415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040a8c50 con 0x7fe4041415c0 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040aa9a0 con 0x7fe4040ae8f0 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040a98a0 con 0x7fe4040aecc0 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40c4e6700 10 monclient(hunting): authenticate will time out at 1263.609546s 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40794/0 (socket says 172.21.15.115:40794) 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48744/0 (socket says 172.21.15.115:48744) 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 -- 172.21.15.115:0/3931333095 learned_addr learned my addr 172.21.15.115:0/3931333095 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): get_auth_request con 0x7fe4041415c0 auth_method 0 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): get_auth_request con 0x7fe4040aecc0 auth_method 0 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:42.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe409a81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40a282700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 0x7fe4040aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40a282700 10 monclient(hunting): get_auth_request con 0x7fe4040ae8f0 auth_method 0 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40a282700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40a282700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40a282700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:42.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient(hunting): handle_auth_done global_id 4758 payload 824 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 -- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 msgr2=0x7fe40413e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 -- 172.21.15.115:0/3931333095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 msgr2=0x7fe4040aa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 0x7fe4040aa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 10 monclient: _finish_hunting 0 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.159+0000 7fe40aa83700 1 monclient: found mon.noname-a 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40aa83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40aa83700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40aa83700 1 -- 172.21.15.115:0/3931333095 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f40094f0 con 0x7fe4041415c0 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe409a81700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:42.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40a282700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 0x7fe4040aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40aa83700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00c960 tx=0x7fe3fc0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 client.?.objecter ms_handle_connect 0x7fe4041415c0 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc012630 con 0x7fe4041415c0 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: dump: 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.179 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 5 monclient: authenticate success, global_id 4758 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 10 monclient: _renew_subs 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc012790 con 0x7fe4041415c0 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4040a9b20 con 0x7fe4041415c0 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc012b50 con 0x7fe4041415c0 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe3fa7fc700 10 monclient: dump: 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:42.181 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe408a7f700 4 set_mon_vals no callback set 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 10 client.4758.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 10 monclient: _renew_subs 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe404151cf0 con 0x7fe4041415c0 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.160+0000 7fe40c4e6700 1 librados: init done 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.161+0000 7fe40c4e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:42.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.161+0000 7fe40c4e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.161+0000 7fe40c4e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.161+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe404072960 con 0x7fe4041415c0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.161+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe3fc01b400 con 0x7fe4041415c0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 1 --2- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 0x7fe3e806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe40a282700 1 --2- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 0x7fe3e806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe3fc0842f0 con 0x7fe4041415c0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 10 client.4758.objecter ms_dispatch 0x7fe404001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 3 client.4758.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 3 client.4758.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe40a282700 10 monclient: get_auth_request con 0x7fe3e8068910 auth_method 0 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 20 client.4758.objecter dump_active .. 0 homeless 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe3fa7fc700 10 client.4758.objecter ms_handle_connect 0x7fe3e8068910 2023-09-30T01:06:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.162+0000 7fe40a282700 1 --2- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 0x7fe3e806adc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40099b0 tx=0x7fe3f4016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:42.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.166+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3fc051b10 con 0x7fe4041415c0 2023-09-30T01:06:42.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.166+0000 7fe3fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:42.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.166+0000 7fe3fa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.393+0000 7fe40c4e6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.393+0000 7fe40c4e6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.393+0000 7fe40c4e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.393+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4040aaae0 con 0x7fe4041415c0 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.394+0000 7fe3fa7fc700 1 -- 172.21.15.115:0/3931333095 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe3fc055130 con 0x7fe4041415c0 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.394+0000 7fe3fa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:42.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.394+0000 7fe3fa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:42.395 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:42.395 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:42.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 10 librados: watch_flush enter 2023-09-30T01:06:42.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 10 librados: watch_flush exit 2023-09-30T01:06:42.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 20 client.4758.objecter shutdown clearing up homeless session... 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 10 client.4758.objecter successfully canceled tick 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 msgr2=0x7fe3e806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 0x7fe3e806adc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40099b0 tx=0x7fe3f4016000 comp rx=0 tx=0).stop 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 10 monclient: shutdown 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 msgr2=0x7fe4041514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00c960 tx=0x7fe3fc0026e0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 shutdown_connections 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> v2:172.21.15.115:6816/115635 conn(0x7fe3e8068910 0x7fe3e806adc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4041415c0 0x7fe4041514c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4040ae8f0 0x7fe4040aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 --2- 172.21.15.115:0/3931333095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe4040aecc0 0x7fe40413e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 >> 172.21.15.115:0/3931333095 conn(0x7fe40406a460 msgr2=0x7fe40413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 shutdown_connections 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 -- 172.21.15.115:0/3931333095 wait complete. 2023-09-30T01:06:42.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:42.396+0000 7fe40c4e6700 1 librados: shutdown 2023-09-30T01:06:45.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:45.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 10 monclient: get_monmap_and_config 2023-09-30T01:06:45.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 10 monclient: build_initial_monmap 2023-09-30T01:06:45.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 10 monclient: monmap: 2023-09-30T01:06:45.537 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:45.538109+0000 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:45.538109+0000 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:45.538 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:45.539 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:45.539 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:45.539 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 1 Processor -- start 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 1 -- start start 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.537+0000 7f3bbfa1a700 10 monclient: init 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-b con 0x7f3bb80ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bb80aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3bb81415c0 0x7f3bb80adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-a con 0x7f3bb81415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80a7040 con 0x7f3bb80aecc0 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb80a57f0 con 0x7f3bb81415c0 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80a5480 con 0x7f3bb80ae8f0 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbfa1a700 10 monclient(hunting): authenticate will time out at 1266.988659s 2023-09-30T01:06:45.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3bb81415c0 0x7f3bb80adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48746/0 (socket says 172.21.15.115:48746) 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 1 -- 172.21.15.115:0/86811207 learned_addr learned my addr 172.21.15.115:0/86811207 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): get_auth_request con 0x7f3bb81415c0 auth_method 0 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): get_auth_request con 0x7f3bb80ae8f0 auth_method 0 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:45.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.538+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_done global_id 4761 payload 274 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 msgr2=0x7f3bb80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/86811207 >> v1:172.21.15.157:6789/0 conn(0x7f3bb80aecc0 legacy=0x7f3bb80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 10 monclient: _finish_hunting 0 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 monclient: found mon.noname-a 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/86811207 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb80a55c0 con 0x7f3bb81415c0 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbd7b6700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbdfb7700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb80adb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009e60 tx=0x7f3bb0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae12580db472db2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 1 -- 172.21.15.115:0/86811207 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000e070 con 0x7f3bb81415c0 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: dump: 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:45.553 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:45.554 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: _finish_auth 0 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 5 monclient: authenticate success, global_id 4761 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 1 -- 172.21.15.115:0/86811207 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb000bb60 con 0x7f3bb81415c0 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:45.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 1 -- 172.21.15.115:0/86811207 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0012430 con 0x7f3bb81415c0 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 4 set_mon_vals no callback set 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3baffff700 10 monclient: dump: 2023-09-30T01:06:45.556 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.557 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 10 monclient: shutdown 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 msgr2=0x7f3bb80adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb80adb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009e60 tx=0x7f3bb0009780 comp rx=0 tx=0).stop 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/86811207 shutdown_connections 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb80adb60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/86811207 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80ae8f0 0x7f3bb80a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.539+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/86811207 >> 172.21.15.115:0/86811207 conn(0x7f3bb806a460 msgr2=0x7f3bb80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:45.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/86811207 shutdown_connections 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/86811207 wait complete. 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 10 monclient: build_initial_monmap 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 10 monclient: monmap: 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:45.559 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 1 librados: starting msgr at 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 1 librados: starting objecter 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.540+0000 7f3bbfa1a700 1 Processor -- start 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 -- start start 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 librados: setting wanted keys 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 librados: calling monclient init 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient: init 2023-09-30T01:06:45.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb80ae8f0 0x7f3bb813e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-b con 0x7f3bb80ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-c con 0x7f3bb80aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): picked mon.noname-a con 0x7f3bb81415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb814eb10 con 0x7f3bb81415c0 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80abae0 con 0x7f3bb80ae8f0 2023-09-30T01:06:45.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb814e810 con 0x7f3bb80aecc0 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbfa1a700 10 monclient(hunting): authenticate will time out at 1266.991734s 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48748/0 (socket says 172.21.15.115:48748) 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40806/0 (socket says 172.21.15.115:40806) 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 1 -- 172.21.15.115:0/600800224 learned_addr learned my addr 172.21.15.115:0/600800224 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): get_auth_request con 0x7f3bb80aecc0 auth_method 0 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:45.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): get_auth_request con 0x7f3bb81415c0 auth_method 0 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbd7b6700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb80ae8f0 0x7f3bb813e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbd7b6700 10 monclient(hunting): get_auth_request con 0x7f3bb80ae8f0 auth_method 0 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbd7b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbd7b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbd7b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:45.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbcfb5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.541+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbd7b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 msgr2=0x7f3bb814e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/600800224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb80ae8f0 msgr2=0x7f3bb813e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb80ae8f0 0x7f3bb813e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 10 monclient: _finish_hunting 0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 monclient: found mon.noname-a 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 -- 172.21.15.115:0/600800224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba80094f0 con 0x7f3bb81415c0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbcfb5700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbdfb7700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3bb806a8b0 tx=0x7f3bb0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 client.?.objecter ms_handle_connect 0x7f3bb81415c0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0012e60 con 0x7f3bb81415c0 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:45.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: dump: 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:45.566 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: _finish_auth 0 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 5 monclient: authenticate success, global_id 4764 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb0027470 con 0x7f3bb81415c0 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 10 monclient: _renew_subs 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb0009a20 con 0x7f3bb81415c0 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:45.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0027810 con 0x7f3bb81415c0 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bad7fa700 10 monclient: dump: 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:45.568 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3baf7fe700 4 set_mon_vals no callback set 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 10 client.4764.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 10 monclient: _renew_subs 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb8151c20 con 0x7f3bb81415c0 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.542+0000 7f3bbfa1a700 1 librados: init done 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.543+0000 7f3bbfa1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.543+0000 7f3bbfa1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.543+0000 7f3bbfa1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.543+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb80a6cf0 con 0x7f3bb81415c0 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3bb0024070 con 0x7f3bb81415c0 2023-09-30T01:06:45.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 1 --2- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 0x7f3b9c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3bb00868f0 con 0x7f3bb81415c0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 10 client.4764.objecter ms_dispatch 0x7f3bb8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 3 client.4764.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 3 client.4764.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bbd7b6700 1 --2- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 0x7f3b9c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bbd7b6700 10 monclient: get_auth_request con 0x7f3b9c068890 auth_method 0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 20 client.4764.objecter dump_active .. 0 homeless 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bad7fa700 10 client.4764.objecter ms_handle_connect 0x7f3b9c068890 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.544+0000 7f3bbd7b6700 1 --2- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 0x7f3b9c06ad40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8006d20 tx=0x7f3ba8009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.548+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bb00541b0 con 0x7f3bb81415c0 2023-09-30T01:06:45.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.548+0000 7f3bad7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:45.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.548+0000 7f3bad7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.769+0000 7f3bbfa1a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.769+0000 7f3bbfa1a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.769+0000 7f3bbfa1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.769+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3bb8072960 con 0x7f3bb81415c0 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.770+0000 7f3bad7fa700 1 -- 172.21.15.115:0/600800224 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3bb00577d0 con 0x7f3bb81415c0 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.770+0000 7f3bad7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:45.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.770+0000 7f3bad7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:45.771 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:45.771 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:45.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.771+0000 7f3bbfa1a700 10 librados: watch_flush enter 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 10 librados: watch_flush exit 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 20 client.4764.objecter shutdown clearing up homeless session... 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 10 client.4764.objecter successfully canceled tick 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 msgr2=0x7f3b9c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 0x7f3b9c06ad40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8006d20 tx=0x7f3ba8009250 comp rx=0 tx=0).stop 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 10 monclient: shutdown 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 msgr2=0x7f3bb81514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3bb806a8b0 tx=0x7f3bb0019000 comp rx=0 tx=0).stop 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 shutdown_connections 2023-09-30T01:06:45.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> v2:172.21.15.115:6816/115635 conn(0x7f3b9c068890 0x7f3b9c06ad40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3bb81415c0 0x7f3bb81514a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb80ae8f0 0x7f3bb813e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 --2- 172.21.15.115:0/600800224 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3bb80aecc0 0x7f3bb814e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 >> 172.21.15.115:0/600800224 conn(0x7f3bb806a460 msgr2=0x7f3bb80ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 shutdown_connections 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 -- 172.21.15.115:0/600800224 wait complete. 2023-09-30T01:06:45.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:45.772+0000 7f3bbfa1a700 1 librados: shutdown 2023-09-30T01:06:48.784 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:48.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.913+0000 7f556304b700 10 monclient: get_monmap_and_config 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.913+0000 7f556304b700 10 monclient: build_initial_monmap 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.913+0000 7f556304b700 10 monclient: monmap: 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:48.914513+0000 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:48.914513+0000 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:48.915 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.914+0000 7f556304b700 1 Processor -- start 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.914+0000 7f556304b700 1 -- start start 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.914+0000 7f556304b700 10 monclient: init 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T01:06:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f555c0ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): picked mon.noname-b con 0x7f555c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f555c1415c0 0x7f555c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): picked mon.noname-c con 0x7f555c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c0a7040 con 0x7f555c0ae8f0 2023-09-30T01:06:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c0a57f0 con 0x7f555c1415c0 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c0a5480 con 0x7f555c0aecc0 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient(hunting): authenticate will time out at 1270.365359s 2023-09-30T01:06:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48752/0 (socket says 172.21.15.115:48752) 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 -- 172.21.15.115:0/1470862294 learned_addr learned my addr 172.21.15.115:0/1470862294 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): get_auth_request con 0x7f555c0aecc0 auth_method 0 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:48.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f55615e8700 1 --2- 172.21.15.115:0/1470862294 >> v2:172.21.15.157:3300/0 conn(0x7f555c1415c0 0x7f555c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f55615e8700 10 monclient(hunting): get_auth_request con 0x7f555c1415c0 auth_method 0 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f55615e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f55615e8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:48.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f55615e8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient(hunting): handle_auth_done global_id 4565 payload 274 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 -- 172.21.15.115:0/1470862294 >> v2:172.21.15.157:3300/0 conn(0x7f555c1415c0 msgr2=0x7f555c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 --2- 172.21.15.115:0/1470862294 >> v2:172.21.15.157:3300/0 conn(0x7f555c1415c0 0x7f555c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 -- 172.21.15.115:0/1470862294 >> v1:172.21.15.157:6789/0 conn(0x7f555c0ae8f0 legacy=0x7f555c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 monclient: found mon.noname-b 2023-09-30T01:06:48.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 -- 172.21.15.115:0/1470862294 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0a55c0 con 0x7f555c0aecc0 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555bfff700 1 --2- 172.21.15.115:0/1470862294 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55480099b0 tx=0x7f5548009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4007d5440466b9ef server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 1 -- 172.21.15.115:0/1470862294 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800b860 con 0x7f555c0aecc0 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: dump: 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.935 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:06:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 1 -- 172.21.15.115:0/1470862294 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554800b9c0 con 0x7f555c0aecc0 2023-09-30T01:06:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 5 monclient: authenticate success, global_id 4565 2023-09-30T01:06:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 1 -- 172.21.15.115:0/1470862294 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800bd60 con 0x7f555c0aecc0 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f556304b700 4 set_mon_vals no callback set 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.915+0000 7f555b7fe700 10 monclient: dump: 2023-09-30T01:06:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 10 monclient: shutdown 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:48.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 -- 172.21.15.115:0/1470862294 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 msgr2=0x7f555c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 --2- 172.21.15.115:0/1470862294 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55480099b0 tx=0x7f5548009250 comp rx=0 tx=0).stop 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 -- 172.21.15.115:0/1470862294 shutdown_connections 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 --2- 172.21.15.115:0/1470862294 >> v2:172.21.15.157:3300/0 conn(0x7f555c1415c0 0x7f555c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 --2- 172.21.15.115:0/1470862294 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a5d10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 -- 172.21.15.115:0/1470862294 >> 172.21.15.115:0/1470862294 conn(0x7f555c06a460 msgr2=0x7f555c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 -- 172.21.15.115:0/1470862294 shutdown_connections 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.916+0000 7f556304b700 1 -- 172.21.15.115:0/1470862294 wait complete. 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 10 monclient: build_initial_monmap 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 10 monclient: monmap: 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:48.940 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 librados: starting msgr at 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 librados: starting objecter 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 Processor -- start 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 -- start start 2023-09-30T01:06:48.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 librados: setting wanted keys 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 1 librados: calling monclient init 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.917+0000 7f556304b700 10 monclient: init 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f555c0ae8f0 0x7f555c13e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): picked mon.noname-b con 0x7f555c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): picked mon.noname-c con 0x7f555c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 0x7f555c0a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): picked mon.noname-a con 0x7f555c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c0ab380 con 0x7f555c1415c0 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c0aa0b0 con 0x7f555c0ae8f0 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c0ab600 con 0x7f555c0aecc0 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f556304b700 10 monclient(hunting): authenticate will time out at 1270.368346s 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 0x7f555c0a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:48760/0 (socket says 172.21.15.115:48760) 2023-09-30T01:06:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 -- 172.21.15.115:0/4260438288 learned_addr learned my addr 172.21.15.115:0/4260438288 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): get_auth_request con 0x7f555c0aecc0 auth_method 0 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): get_auth_request con 0x7f555c1415c0 auth_method 0 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f555c0ae8f0 0x7f555c13e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): get_auth_request con 0x7f555c0ae8f0 auth_method 0 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5560de7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient(hunting): handle_auth_done global_id 4568 payload 824 2023-09-30T01:06:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 -- 172.21.15.115:0/4260438288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f555c0ae8f0 msgr2=0x7f555c13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f555c0ae8f0 0x7f555c13e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 -- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 msgr2=0x7f555c0a7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 0x7f555c0a7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 monclient: found mon.noname-c 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 -- 172.21.15.115:0/4260438288 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480094f0 con 0x7f555c0aecc0 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f55615e8700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 0x7f555c0a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f555bfff700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f555c0b0920 tx=0x7f5548017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5558ff9700 10 client.?.objecter ms_handle_connect 0x7f555c0aecc0 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.918+0000 7f5558ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548011e40 con 0x7f555c0aecc0 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: dump: 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: _finish_auth 0 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548024440 con 0x7f555c0aecc0 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 5 monclient: authenticate success, global_id 4568 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 monclient: _renew_subs 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c0ab8c0 con 0x7f555c0aecc0 2023-09-30T01:06:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548024800 con 0x7f555c0aecc0 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f5558ff9700 10 monclient: dump: 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:48.949 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f555affd700 4 set_mon_vals no callback set 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 monclient: _renew_subs 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c0a9910 con 0x7f555c0aecc0 2023-09-30T01:06:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 1 librados: init done 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.919+0000 7f556304b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f556304b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f556304b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c072960 con 0x7f555c0aecc0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f554801a420 con 0x7f555c0aecc0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 1 --2- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 0x7f554006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5548085d50 con 0x7f555c0aecc0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 10 client.4568.objecter ms_dispatch 0x7f555c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 3 client.4568.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5558ff9700 3 client.4568.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5560de7700 1 --2- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 0x7f554006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.920+0000 7f5560de7700 10 monclient: get_auth_request con 0x7f55400688c0 auth_method 0 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.921+0000 7f5558ff9700 20 client.4568.objecter dump_active .. 0 homeless 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.921+0000 7f5558ff9700 10 client.4568.objecter ms_handle_connect 0x7f55400688c0 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.921+0000 7f5560de7700 1 --2- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 0x7f554006ad70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5550003a80 tx=0x7f5550007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.925+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5548089030 con 0x7f555c0aecc0 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.925+0000 7f5558ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:48.925+0000 7f5558ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f556304b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f556304b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f556304b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f555c141a40 con 0x7f555c0aecc0 2023-09-30T01:06:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f5558ff9700 1 -- 172.21.15.115:0/4260438288 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5548053610 con 0x7f555c0aecc0 2023-09-30T01:06:49.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f5558ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:49.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.150+0000 7f5558ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:49.151 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:49.151 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 10 librados: watch_flush enter 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 10 librados: watch_flush exit 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 20 client.4568.objecter shutdown clearing up homeless session... 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 10 client.4568.objecter successfully canceled tick 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 msgr2=0x7f554006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 0x7f554006ad70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5550003a80 tx=0x7f5550007ec0 comp rx=0 tx=0).stop 2023-09-30T01:06:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 10 monclient: shutdown 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 msgr2=0x7f555c0a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f555c0b0920 tx=0x7f5548017000 comp rx=0 tx=0).stop 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 shutdown_connections 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> v2:172.21.15.115:6816/115635 conn(0x7f55400688c0 0x7f554006ad70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f555c1415c0 0x7f555c0a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f555c0ae8f0 0x7f555c13e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:49.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 --2- 172.21.15.115:0/4260438288 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f555c0aecc0 0x7f555c0a7760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:49.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 >> 172.21.15.115:0/4260438288 conn(0x7f555c06a460 msgr2=0x7f555c144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:49.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 shutdown_connections 2023-09-30T01:06:49.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 -- 172.21.15.115:0/4260438288 wait complete. 2023-09-30T01:06:49.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:49.152+0000 7f556304b700 1 librados: shutdown 2023-09-30T01:06:52.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.292+0000 7fd0d29a7700 10 monclient: get_monmap_and_config 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.292+0000 7fd0d29a7700 10 monclient: build_initial_monmap 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.293+0000 7fd0d29a7700 10 monclient: monmap: 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:52.293962+0000 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:52.293962+0000 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:52.293 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:52.294 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.293+0000 7fd0d29a7700 1 Processor -- start 2023-09-30T01:06:52.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.293+0000 7fd0d29a7700 1 -- start start 2023-09-30T01:06:52.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.293+0000 7fd0d29a7700 10 monclient: init 2023-09-30T01:06:52.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-a con 0x7fd0cc0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0cc0aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0cc1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc0a7040 con 0x7fd0cc0aecc0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc0a57f0 con 0x7fd0cc1415c0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc0a5480 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0d29a7700 10 monclient(hunting): authenticate will time out at 1273.744491s 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33540/0 (socket says 172.21.15.115:33540) 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 -- 172.21.15.115:0/293989027 learned_addr learned my addr 172.21.15.115:0/293989027 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cb7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd0cc0aecc0 0x7fd0cc0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:60312/0 (socket says 172.21.15.115:60312) 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): get_auth_request con 0x7fd0cc0ae8f0 auth_method 0 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:52.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_done global_id 4770 payload 274 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 -- 172.21.15.115:0/293989027 >> v1:172.21.15.157:6789/0 conn(0x7fd0cc1415c0 legacy=0x7fd0cc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 -- 172.21.15.115:0/293989027 >> v1:172.21.15.115:6789/0 conn(0x7fd0cc0aecc0 legacy=0x7fd0cc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 monclient: found mon.noname-a 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0caffd700 1 -- 172.21.15.115:0/293989027 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374998575 0 0) 0x7fd0cc0a7040 con 0x7fd0cc0aecc0 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0cbfff700 1 -- 172.21.15.115:0/293989027 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc0a55c0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.294+0000 7fd0caffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0cbfff700 1 --2- 172.21.15.115:0/293989027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00099b0 tx=0x7fd0c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbe85948b6ef2d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:52.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 1 -- 172.21.15.115:0/293989027 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000b820 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: dump: 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:52.305 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: _finish_auth 0 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 5 monclient: authenticate success, global_id 4770 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 1 -- 172.21.15.115:0/293989027 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c000b980 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 4 set_mon_vals no callback set 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 1 -- 172.21.15.115:0/293989027 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000bd20 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:52.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0caffd700 10 monclient: dump: 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 10 monclient: shutdown 2023-09-30T01:06:52.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 -- 172.21.15.115:0/293989027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 msgr2=0x7fd0cc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/293989027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00099b0 tx=0x7fd0c0009250 comp rx=0 tx=0).stop 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 -- 172.21.15.115:0/293989027 shutdown_connections 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/293989027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0a5d10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 -- 172.21.15.115:0/293989027 >> 172.21.15.115:0/293989027 conn(0x7fd0cc06a460 msgr2=0x7fd0cc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 -- 172.21.15.115:0/293989027 shutdown_connections 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.295+0000 7fd0d29a7700 1 -- 172.21.15.115:0/293989027 wait complete. 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.296+0000 7fd0d29a7700 10 monclient: build_initial_monmap 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.296+0000 7fd0d29a7700 10 monclient: monmap: 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:52.308 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.296+0000 7fd0d29a7700 1 librados: starting msgr at 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.296+0000 7fd0d29a7700 1 librados: starting objecter 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 Processor -- start 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 -- start start 2023-09-30T01:06:52.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 librados: setting wanted keys 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 librados: calling monclient init 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient: init 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-a con 0x7fd0cc0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0cc0aecc0 0x7fd0cc13e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-b con 0x7fd0cc0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd0cc1415c0 0x7fd0cc1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): picked mon.noname-c con 0x7fd0cc1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc0a8c50 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc0aa9a0 con 0x7fd0cc0aecc0 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc0a98a0 con 0x7fd0cc1415c0 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d29a7700 10 monclient(hunting): authenticate will time out at 1273.747846s 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33554/0 (socket says 172.21.15.115:33554) 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 1 -- 172.21.15.115:0/4154322995 learned_addr learned my addr 172.21.15.115:0/4154322995 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 10 monclient(hunting): get_auth_request con 0x7fd0cc0ae8f0 auth_method 0 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:52.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.297+0000 7fd0d0f44700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd0cc1415c0 0x7fd0cc1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): get_auth_request con 0x7fd0cc1415c0 auth_method 0 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d0f44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cb7fe700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0cc0aecc0 0x7fd0cc13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:52.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient(hunting): handle_auth_done global_id 4776 payload 824 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 -- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd0cc1415c0 msgr2=0x7fd0cc1514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd0cc1415c0 0x7fd0cc1514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 -- 172.21.15.115:0/4154322995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0cc0aecc0 msgr2=0x7fd0cc13e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0cc0aecc0 0x7fd0cc13e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 monclient: found mon.noname-a 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 -- 172.21.15.115:0/4154322995 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c00094f0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0cbfff700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0b0920 tx=0x7fd0c0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 client.?.objecter ms_handle_connect 0x7fd0cc0ae8f0 2023-09-30T01:06:52.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000bbc0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: dump: 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:52.314 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: _finish_auth 0 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c0020070 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 5 monclient: authenticate success, global_id 4776 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 10 monclient: _renew_subs 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc0a9ac0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c001c6a0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0b3fff700 10 monclient: dump: 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:52.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0ca7fc700 4 set_mon_vals no callback set 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 10 client.4776.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 10 monclient: _renew_subs 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc151c40 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.298+0000 7fd0d29a7700 1 librados: init done 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.299+0000 7fd0d29a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.299+0000 7fd0d29a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.299+0000 7fd0d29a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.299+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc0aac80 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.299+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd0c0022030 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 1 --2- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 0x7fd0b806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:52.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd0c0085d00 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 10 client.4776.objecter ms_dispatch 0x7fd0cc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 3 client.4776.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 3 client.4776.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0cb7fe700 1 --2- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 0x7fd0b806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0cb7fe700 10 monclient: get_auth_request con 0x7fd0b8068890 auth_method 0 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 20 client.4776.objecter dump_active .. 0 homeless 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0b3fff700 10 client.4776.objecter ms_handle_connect 0x7fd0b8068890 2023-09-30T01:06:52.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.300+0000 7fd0cb7fe700 1 --2- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 0x7fd0b806ad40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc007fd0 tx=0x7fd0bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:52.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.304+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0c00535c0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.304+0000 7fd0b3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:52.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.304+0000 7fd0b3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.525+0000 7fd0d29a7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.525+0000 7fd0d29a7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.525+0000 7fd0d29a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.525+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd0cc0aac80 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.526+0000 7fd0b3fff700 1 -- 172.21.15.115:0/4154322995 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd0c0056be0 con 0x7fd0cc0ae8f0 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.526+0000 7fd0b3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:52.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.526+0000 7fd0b3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:52.527 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:52.527 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:52.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 10 librados: watch_flush enter 2023-09-30T01:06:52.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 10 librados: watch_flush exit 2023-09-30T01:06:52.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 20 client.4776.objecter shutdown clearing up homeless session... 2023-09-30T01:06:52.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 10 client.4776.objecter successfully canceled tick 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 msgr2=0x7fd0b806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 0x7fd0b806ad40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc007fd0 tx=0x7fd0bc009040 comp rx=0 tx=0).stop 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 10 monclient: shutdown 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 msgr2=0x7fd0cc0aa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0b0920 tx=0x7fd0c0016000 comp rx=0 tx=0).stop 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 shutdown_connections 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> v2:172.21.15.115:6816/115635 conn(0x7fd0b8068890 0x7fd0b806ad40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd0cc1415c0 0x7fd0cc1514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0cc0ae8f0 0x7fd0cc0aa2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 --2- 172.21.15.115:0/4154322995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0cc0aecc0 0x7fd0cc13e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 >> 172.21.15.115:0/4154322995 conn(0x7fd0cc06a460 msgr2=0x7fd0cc13fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 shutdown_connections 2023-09-30T01:06:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 -- 172.21.15.115:0/4154322995 wait complete. 2023-09-30T01:06:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:52.528+0000 7fd0d29a7700 1 librados: shutdown 2023-09-30T01:06:55.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:55.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.668+0000 7f5fef4e4700 10 monclient: get_monmap_and_config 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.668+0000 7f5fef4e4700 10 monclient: build_initial_monmap 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.668+0000 7f5fef4e4700 10 monclient: monmap: 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:55.669823+0000 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:55.669823+0000 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:55.669 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:55.670 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:55.670 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:55.670 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:55.670 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.669+0000 7f5fef4e4700 1 Processor -- start 2023-09-30T01:06:55.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.669+0000 7f5fef4e4700 1 -- start start 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.669+0000 7f5fef4e4700 10 monclient: init 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 0x7f5fe80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-b con 0x7f5fe80ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5fe80aecc0 0x7f5fe80b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-a con 0x7f5fe80aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fe81415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe80a7040 con 0x7f5fe81415c0 2023-09-30T01:06:55.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe80b19b0 con 0x7f5fe80aecc0 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80a57f0 con 0x7f5fe80ae8f0 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fef4e4700 10 monclient(hunting): authenticate will time out at 1277.120302s 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 0x7f5fe80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5fe80aecc0 0x7f5fe80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f5fe80aecc0 0x7f5fe80b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33558/0 (socket says 172.21.15.115:33558) 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 -- 172.21.15.115:0/769748310 learned_addr learned my addr 172.21.15.115:0/769748310 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:55.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 0x7f5fe80a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34960/0 (socket says 172.21.15.115:34960) 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): get_auth_request con 0x7f5fe80aecc0 auth_method 0 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): get_auth_request con 0x7f5fe80ae8f0 auth_method 0 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:55.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5fed280700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient(hunting): handle_auth_done global_id 4779 payload 274 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 -- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 msgr2=0x7f5fe80a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 --2- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 0x7f5fe80a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 -- 172.21.15.115:0/769748310 >> v1:172.21.15.157:6789/0 conn(0x7f5fe81415c0 legacy=0x7f5fe80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient: _finish_hunting 0 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 monclient: found mon.noname-a 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 -- 172.21.15.115:0/769748310 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80a5480 con 0x7f5fe80aecc0 2023-09-30T01:06:55.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.670+0000 7f5feca7f700 1 --2- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe80aecc0 0x7f5fe80b1470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40099b0 tx=0x7f5fe4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b5810da9838805 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 1 -- 172.21.15.115:0/769748310 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4009ea0 con 0x7f5fe80aecc0 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: dump: 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.682 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: _finish_auth 0 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 5 monclient: authenticate success, global_id 4779 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 1 -- 172.21.15.115:0/769748310 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400b820 con 0x7f5fe80aecc0 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 4 set_mon_vals no callback set 2023-09-30T01:06:55.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 1 -- 172.21.15.115:0/769748310 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400bbc0 con 0x7f5fe80aecc0 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fdffff700 10 monclient: dump: 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:55.684 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 10 monclient: shutdown 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 -- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe80aecc0 msgr2=0x7f5fe80b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe80aecc0 0x7f5fe80b1470 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40099b0 tx=0x7f5fe4009250 comp rx=0 tx=0).stop 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 -- 172.21.15.115:0/769748310 shutdown_connections 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80ae8f0 0x7f5fe80a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/769748310 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe80aecc0 0x7f5fe80b1470 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 -- 172.21.15.115:0/769748310 >> 172.21.15.115:0/769748310 conn(0x7f5fe806a460 msgr2=0x7f5fe80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 -- 172.21.15.115:0/769748310 shutdown_connections 2023-09-30T01:06:55.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.671+0000 7f5fef4e4700 1 -- 172.21.15.115:0/769748310 wait complete. 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 10 monclient: build_initial_monmap 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 10 monclient: monmap: 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:55.686 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 librados: starting msgr at 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 librados: starting objecter 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 Processor -- start 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 -- start start 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 librados: setting wanted keys 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 1 librados: calling monclient init 2023-09-30T01:06:55.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.672+0000 7f5fef4e4700 10 monclient: init 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe80ae8f0 0x7f5fe80a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-b con 0x7f5fe80ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-c con 0x7f5fe80aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 0x7f5fe80aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): picked mon.noname-a con 0x7f5fe81415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:55.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80ad770 con 0x7f5fe81415c0 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe813fb70 con 0x7f5fe80ae8f0 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80a8870 con 0x7f5fe80aecc0 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fef4e4700 10 monclient(hunting): authenticate will time out at 1277.123427s 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 0x7f5fe80aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34962/0 (socket says 172.21.15.115:34962) 2023-09-30T01:06:55.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 1 -- 172.21.15.115:0/2117814911 learned_addr learned my addr 172.21.15.115:0/2117814911 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): get_auth_request con 0x7f5fe81415c0 auth_method 0 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): get_auth_request con 0x7f5fe80aecc0 auth_method 0 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:55.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient(hunting): handle_auth_done global_id 4782 payload 824 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fed280700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe80ae8f0 0x7f5fe80a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 -- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 msgr2=0x7f5fe813f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 -- 172.21.15.115:0/2117814911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe80ae8f0 msgr2=0x7f5fe80a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe80ae8f0 0x7f5fe80a96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient: _finish_hunting 0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 monclient: found mon.noname-a 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 -- 172.21.15.115:0/2117814911 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40094f0 con 0x7f5fe81415c0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feca7f700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5feda81700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 0x7f5fe80aa190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000bad0 tx=0x7f5fe0009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fdd7fa700 10 client.?.objecter ms_handle_connect 0x7f5fe81415c0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.673+0000 7f5fdd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000c840 con 0x7f5fe81415c0 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:55.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: dump: 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:55.693 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: _finish_auth 0 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe000c9a0 con 0x7f5fe81415c0 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 5 monclient: authenticate success, global_id 4782 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 10 monclient: _renew_subs 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000cd60 con 0x7f5fe81415c0 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe8151030 con 0x7f5fe81415c0 2023-09-30T01:06:55.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdf7fe700 4 set_mon_vals no callback set 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fdd7fa700 10 monclient: dump: 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:55.695 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 10 client.4782.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 10 monclient: _renew_subs 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe80aa6d0 con 0x7f5fe81415c0 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.674+0000 7f5fef4e4700 1 librados: init done 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.675+0000 7f5fef4e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.675+0000 7f5fef4e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.675+0000 7f5fef4e4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.675+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80a9eb0 con 0x7f5fe81415c0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5fe0014620 con 0x7f5fe81415c0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 1 --2- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 0x7f5fcc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5fe0084420 con 0x7f5fe81415c0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 10 client.4782.objecter ms_dispatch 0x7f5fe8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 3 client.4782.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 3 client.4782.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fed280700 1 --2- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 0x7f5fcc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fed280700 10 monclient: get_auth_request con 0x7f5fcc068890 auth_method 0 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 20 client.4782.objecter dump_active .. 0 homeless 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fdd7fa700 10 client.4782.objecter ms_handle_connect 0x7f5fcc068890 2023-09-30T01:06:55.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.676+0000 7f5fed280700 1 --2- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 0x7f5fcc06ad40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80093f0 tx=0x7f5fd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:55.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.680+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fe0051d60 con 0x7f5fe81415c0 2023-09-30T01:06:55.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.680+0000 7f5fdd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:55.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.680+0000 7f5fdd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:55.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fef4e4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:55.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fef4e4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:55.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fef4e4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:06:55.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5fe80a9eb0 con 0x7f5fe81415c0 2023-09-30T01:06:55.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fdd7fa700 1 -- 172.21.15.115:0/2117814911 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5fe80a9eb0 con 0x7f5fe81415c0 2023-09-30T01:06:55.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fdd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:55.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.901+0000 7f5fdd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:55.902 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 10 librados: watch_flush enter 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 10 librados: watch_flush exit 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 20 client.4782.objecter shutdown clearing up homeless session... 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 10 client.4782.objecter successfully canceled tick 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 msgr2=0x7f5fcc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 0x7f5fcc06ad40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80093f0 tx=0x7f5fd8008040 comp rx=0 tx=0).stop 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 10 monclient: shutdown 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 msgr2=0x7f5fe80aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:55.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 0x7f5fe80aa190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000bad0 tx=0x7f5fe0009760 comp rx=0 tx=0).stop 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 shutdown_connections 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> v2:172.21.15.115:6816/115635 conn(0x7f5fcc068890 0x7f5fcc06ad40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5fe81415c0 0x7f5fe80aa190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe80ae8f0 0x7f5fe80a96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 --2- 172.21.15.115:0/2117814911 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f5fe80aecc0 0x7f5fe813f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 >> 172.21.15.115:0/2117814911 conn(0x7f5fe806a460 msgr2=0x7f5fe813abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 shutdown_connections 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 -- 172.21.15.115:0/2117814911 wait complete. 2023-09-30T01:06:55.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:55.903+0000 7f5fef4e4700 1 librados: shutdown 2023-09-30T01:06:58.916 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 10 monclient: get_monmap_and_config 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 10 monclient: build_initial_monmap 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 10 monclient: monmap: 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.045 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:06:59.046025+0000 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:06:59.046025+0000 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:06:59.046 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:06:59.047 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:06:59.047 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:06:59.047 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 1 Processor -- start 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 1 -- start start 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.045+0000 7f3517150700 10 monclient: init 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35100ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f35100aecc0 0x7f35100a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): picked mon.noname-c con 0x7f35100aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35101415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:06:59.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100a7040 con 0x7f35100ae8f0 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100a57f0 con 0x7f35101415c0 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35100a5480 con 0x7f35100aecc0 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3517150700 10 monclient(hunting): authenticate will time out at 1280.496560s 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3514eec700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f35100ae8f0 0x7f35100b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:60324/0 (socket says 172.21.15.115:60324) 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f3514eec700 1 -- 172.21.15.115:0/2168047240 learned_addr learned my addr 172.21.15.115:0/2168047240 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 1 --2- 172.21.15.115:0/2168047240 >> v2:172.21.15.157:3300/0 conn(0x7f35100aecc0 0x7f35100a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 10 monclient(hunting): get_auth_request con 0x7f35100aecc0 auth_method 0 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249093915 0 0) 0x7f35100a7040 con 0x7f35100ae8f0 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350f7fe700 10 monclient(hunting): my global_id is 4785 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500003570 con 0x7f35100ae8f0 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.046+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027160495 0 0) 0x7f35100a57f0 con 0x7f35101415c0 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): my global_id is 4663 2023-09-30T01:06:59.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35100a7040 con 0x7f35101415c0 2023-09-30T01:06:59.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735262429 0 0) 0x7f3500003570 con 0x7f35100ae8f0 2023-09-30T01:06:59.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 >> v2:172.21.15.157:3300/0 conn(0x7f35100aecc0 msgr2=0x7f35100a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:59.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 --2- 172.21.15.115:0/2168047240 >> v2:172.21.15.157:3300/0 conn(0x7f35100aecc0 0x7f35100a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 >> v1:172.21.15.157:6789/0 conn(0x7f35101415c0 legacy=0x7f35100adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100a55c0 con 0x7f35100ae8f0 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3517150700 5 monclient: authenticate success, global_id 4785 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f3504001470 con 0x7f35100ae8f0 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3517150700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: dump: 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:59.066 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 >> v1:172.21.15.115:6789/0 conn(0x7f35100ae8f0 legacy=0x7f35100b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 0x7f3500003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): picked mon.c con 0x7f3500003bf0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3500004810 0x7f3500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): picked mon.a con 0x7f3500004810 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35000051c0 0x7f350000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): picked mon.b con 0x7f35000051c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100a5480 con 0x7f3500004810 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350000c0b0 con 0x7f35000051c0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350000c270 con 0x7f3500003bf0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3517150700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 0x7f3500003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3500004810 0x7f3500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): get_auth_request con 0x7f3500003bf0 auth_method 0 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): get_auth_request con 0x7f3500004810 auth_method 0 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35000051c0 0x7f350000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): get_auth_request con 0x7f35000051c0 auth_method 0 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient(hunting): handle_auth_done global_id 4785 payload 293 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 -- 172.21.15.115:0/2168047240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35000051c0 msgr2=0x7f350000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35000051c0 0x7f350000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 -- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3500004810 msgr2=0x7f3500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3500004810 0x7f3500004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient: _finish_hunting 0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 monclient: found mon.c 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f35156ed700 1 -- 172.21.15.115:0/2168047240 reap_dead start 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 -- 172.21.15.115:0/2168047240 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500003570 con 0x7f3500003bf0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 10 monclient: _finish_auth 0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350ffff700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3500004810 0x7f3500004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f3514eec700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 0x7f3500003fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f350400c2d0 tx=0x7f3504001e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0fad28ee2427b01 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400cbc0 con 0x7f3500003bf0 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.047+0000 7f350f7fe700 10 monclient: dump: 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:06:59.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400cd20 con 0x7f3500003bf0 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 10 monclient: get_monmap_and_config success 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 4 set_mon_vals no callback set 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 1 -- 172.21.15.115:0/2168047240 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3504005b70 con 0x7f3500003bf0 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f350f7fe700 10 monclient: dump: 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:59.073 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.074 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.074 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.074 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:59.074 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.074 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 10 monclient: shutdown 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:59.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 -- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 msgr2=0x7f3500003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 0x7f3500003fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f350400c2d0 tx=0x7f3504001e10 comp rx=0 tx=0).stop 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 -- 172.21.15.115:0/2168047240 shutdown_connections 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 --2- 172.21.15.115:0/2168047240 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3500003bf0 0x7f3500003fe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 -- 172.21.15.115:0/2168047240 >> 172.21.15.115:0/2168047240 conn(0x7f351006a460 msgr2=0x7f35100630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 -- 172.21.15.115:0/2168047240 shutdown_connections 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.048+0000 7f3517150700 1 -- 172.21.15.115:0/2168047240 wait complete. 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.049+0000 7f3517150700 10 monclient: build_initial_monmap 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.049+0000 7f3517150700 10 monclient: monmap: 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:06:59.076 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.049+0000 7f3517150700 1 librados: starting msgr at 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.049+0000 7f3517150700 1 librados: starting objecter 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.049+0000 7f3517150700 1 Processor -- start 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 -- start start 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 librados: setting wanted keys 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 librados: calling monclient init 2023-09-30T01:06:59.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient: init 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient: _reopen_session rank -1 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3500003bf0 0x7f351013e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): picked mon.noname-b con 0x7f3500003bf0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): picked mon.noname-c con 0x7f35100a9760 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): picked mon.noname-a con 0x7f351013ab70 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100aa760 con 0x7f351013ab70 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100aaa10 con 0x7f3500003bf0 2023-09-30T01:06:59.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): start opening mon connection 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351013b520 con 0x7f35100a9760 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): _renew_subs 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f3517150700 10 monclient(hunting): authenticate will time out at 1280.500866s 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:34976/0 (socket says 172.21.15.115:34976) 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33578/0 (socket says 172.21.15.115:33578) 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 1 -- 172.21.15.115:0/2249352905 learned_addr learned my addr 172.21.15.115:0/2249352905 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 10 monclient(hunting): get_auth_request con 0x7f35100a9760 auth_method 0 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f350ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 10 monclient(hunting): get_auth_request con 0x7f351013ab70 auth_method 0 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.050+0000 7f35156ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f35156ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3514eec700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3500003bf0 0x7f351013e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3514eec700 10 monclient(hunting): get_auth_request con 0x7f3500003bf0 auth_method 0 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3514eec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3514eec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3514eec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 -- 172.21.15.115:0/2249352905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3500003bf0 msgr2=0x7f351013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3500003bf0 0x7f351013e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 -- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 msgr2=0x7f351013afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 monclient: found mon.noname-c 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 -- 172.21.15.115:0/2249352905 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510151590 con 0x7f35100a9760 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f35156ed700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350ffff700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f34fc011d50 tx=0x7f34fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 client.?.objecter ms_handle_connect 0x7f35100a9760 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc011e50 con 0x7f35100a9760 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: dump: 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.082 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: _finish_auth 0 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc00fcd0 con 0x7f35100a9760 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 5 monclient: authenticate success, global_id 4580 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc01d6b0 con 0x7f35100a9760 2023-09-30T01:06:59.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 10 monclient: _renew_subs 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101517e0 con 0x7f35100a9760 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350effd700 4 set_mon_vals no callback set 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f350cff9700 10 monclient: dump: 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:06:59.084 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 10 monclient: _renew_subs 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3510151d20 con 0x7f35100a9760 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.051+0000 7f3517150700 1 librados: init done 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.052+0000 7f3517150700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.052+0000 7f3517150700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.052+0000 7f3517150700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.052+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3510072960 con 0x7f35100a9760 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f34fc01daa0 con 0x7f35100a9760 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 1 --2- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 0x7f34f406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f34fc018070 con 0x7f35100a9760 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 10 client.4580.objecter ms_dispatch 0x7f3510001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 3 client.4580.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 3 client.4580.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f3514eec700 1 --2- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 0x7f34f406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f3514eec700 10 monclient: get_auth_request con 0x7f34f4068890 auth_method 0 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 20 client.4580.objecter dump_active .. 0 homeless 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f350cff9700 10 client.4580.objecter ms_handle_connect 0x7f34f4068890 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.053+0000 7f3514eec700 1 --2- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 0x7f34f406ad40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3504008fd0 tx=0x7f3504015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:06:59.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.057+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34fc0525b0 con 0x7f35100a9760 2023-09-30T01:06:59.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.057+0000 7f350cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:06:59.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.057+0000 7f350cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:06:59.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f3517150700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:06:59.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f3517150700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f3517150700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f35100aab50 con 0x7f35100a9760 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f350cff9700 1 -- 172.21.15.115:0/2249352905 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f34fc022080 con 0x7f35100a9760 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f350cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.283+0000 7f350cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:06:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:59.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 10 librados: watch_flush enter 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 10 librados: watch_flush exit 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 20 client.4580.objecter shutdown clearing up homeless session... 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 10 client.4580.objecter successfully canceled tick 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 msgr2=0x7f34f406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 0x7f34f406ad40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3504008fd0 tx=0x7f3504015000 comp rx=0 tx=0).stop 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 10 monclient: shutdown 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 msgr2=0x7f35100aa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f34fc011d50 tx=0x7f34fc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 shutdown_connections 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> v2:172.21.15.115:6816/115635 conn(0x7f34f4068890 0x7f34f406ad40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f351013ab70 0x7f351013afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3500003bf0 0x7f351013e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 --2- 172.21.15.115:0/2249352905 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35100a9760 0x7f35100aa220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.285+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 >> 172.21.15.115:0/2249352905 conn(0x7f351006a460 msgr2=0x7f35101409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.286+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 shutdown_connections 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.286+0000 7f3517150700 1 -- 172.21.15.115:0/2249352905 wait complete. 2023-09-30T01:06:59.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:06:59.286+0000 7f3517150700 1 librados: shutdown 2023-09-30T01:07:02.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:02.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.430+0000 7f0211dd0700 10 monclient: get_monmap_and_config 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.430+0000 7f0211dd0700 10 monclient: build_initial_monmap 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.430+0000 7f0211dd0700 10 monclient: monmap: 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:02.431826+0000 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:02.431826+0000 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:02.443 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.431+0000 7f0211dd0700 1 Processor -- start 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.431+0000 7f0211dd0700 1 -- start start 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.431+0000 7f0211dd0700 10 monclient: init 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 0x7f020c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-b con 0x7f020c0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:02.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f020c0aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 0x7f020c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-c con 0x7f020c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c0a7040 con 0x7f020c0aecc0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020c0a57f0 con 0x7f020c1415c0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0a5480 con 0x7f020c0ae8f0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f0211dd0700 10 monclient(hunting): authenticate will time out at 1283.882297s 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 0x7f020c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020affd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f020c0aecc0 0x7f020c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33002/0 (socket says 172.21.15.115:33002) 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020affd700 1 -- 172.21.15.115:0/1182085085 learned_addr learned my addr 172.21.15.115:0/1182085085 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): get_auth_request con 0x7f020c0ae8f0 auth_method 0 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient(hunting): handle_auth_done global_id 4583 payload 274 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 -- 172.21.15.115:0/1182085085 >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 msgr2=0x7f020c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020bfff700 1 --2- 172.21.15.115:0/1182085085 >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 0x7f020c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020a7fc700 1 -- 172.21.15.115:0/1182085085 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057888876 0 0) 0x7f020c0a7040 con 0x7f020c0aecc0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 --2- 172.21.15.115:0/1182085085 >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 0x7f020c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 -- 172.21.15.115:0/1182085085 >> v1:172.21.15.115:6789/0 conn(0x7f020c0aecc0 legacy=0x7f020c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 -- 172.21.15.115:0/1182085085 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0a55c0 con 0x7f020c0ae8f0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020bfff700 10 monclient: get_auth_request con 0x7f020c1415c0 auth_method 0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020bfff700 1 --2- 172.21.15.115:0/1182085085 >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 0x7f020c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.432+0000 7f020b7fe700 1 --2- 172.21.15.115:0/1182085085 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 0x7f020c0a5d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f02000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a4b3ca459c97abd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:02.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 1 -- 172.21.15.115:0/1182085085 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200009e80 con 0x7f020c0ae8f0 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: dump: 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:02.448 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 1 -- 172.21.15.115:0/1182085085 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020000b860 con 0x7f020c0ae8f0 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 5 monclient: authenticate success, global_id 4583 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 1 -- 172.21.15.115:0/1182085085 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020000bbe0 con 0x7f020c0ae8f0 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 4 set_mon_vals no callback set 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f020a7fc700 10 monclient: dump: 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:02.450 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 10 monclient: shutdown 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 -- 172.21.15.115:0/1182085085 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 msgr2=0x7f020c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 --2- 172.21.15.115:0/1182085085 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 0x7f020c0a5d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f02000091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 -- 172.21.15.115:0/1182085085 shutdown_connections 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 --2- 172.21.15.115:0/1182085085 >> v2:172.21.15.157:3300/0 conn(0x7f020c1415c0 0x7f020c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 --2- 172.21.15.115:0/1182085085 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c0ae8f0 0x7f020c0a5d10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 -- 172.21.15.115:0/1182085085 >> 172.21.15.115:0/1182085085 conn(0x7f020c06a460 msgr2=0x7f020c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 -- 172.21.15.115:0/1182085085 shutdown_connections 2023-09-30T01:07:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.433+0000 7f0211dd0700 1 -- 172.21.15.115:0/1182085085 wait complete. 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 10 monclient: build_initial_monmap 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 10 monclient: monmap: 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:02.452 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 librados: starting msgr at 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 librados: starting objecter 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 Processor -- start 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 -- start start 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 librados: setting wanted keys 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 1 librados: calling monclient init 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.434+0000 7f0211dd0700 10 monclient: init 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:02.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-a con 0x7f020c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 0x7f020c0a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-b con 0x7f020c0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 0x7f020c0a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): picked mon.noname-c con 0x7f020c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0ab690 con 0x7f020c0ae8f0 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:02.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0a9500 con 0x7f020c0aecc0 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c1402d0 con 0x7f020c1415c0 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f0211dd0700 10 monclient(hunting): authenticate will time out at 1283.885377s 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50342/0 (socket says 172.21.15.115:50342) 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 -- 172.21.15.115:0/2532502583 learned_addr learned my addr 172.21.15.115:0/2532502583 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): get_auth_request con 0x7f020c0ae8f0 auth_method 0 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:02.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 0x7f020c0a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): get_auth_request con 0x7f020c1415c0 auth_method 0 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:02.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020affd700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 0x7f020c0a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient(hunting): handle_auth_done global_id 4794 payload 824 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 -- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 msgr2=0x7f020c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 0x7f020c0a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 -- 172.21.15.115:0/2532502583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 msgr2=0x7f020c0a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 0x7f020c0a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 -- 172.21.15.115:0/2532502583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0200009420 con 0x7f020c0ae8f0 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020affd700 10 monclient: get_auth_request con 0x7f020c0aecc0 auth_method 0 2023-09-30T01:07:02.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020bfff700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 0x7f020c0a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020affd700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 0x7f020c0a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f020b7fe700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f020c0b0920 tx=0x7f0200017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f01f37fe700 10 client.?.objecter ms_handle_connect 0x7f020c0ae8f0 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.435+0000 7f01f37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020000bbe0 con 0x7f020c0ae8f0 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: dump: 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: _finish_auth 0 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200011420 con 0x7f020c0ae8f0 2023-09-30T01:07:02.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 5 monclient: authenticate success, global_id 4794 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 10 monclient: _renew_subs 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c0aa210 con 0x7f020c0ae8f0 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02000117e0 con 0x7f020c0ae8f0 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f01f37fe700 10 monclient: dump: 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.460 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0209ffb700 4 set_mon_vals no callback set 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 10 client.4794.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 10 monclient: _renew_subs 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c150c50 con 0x7f020c0ae8f0 2023-09-30T01:07:02.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.436+0000 7f0211dd0700 1 librados: init done 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f0211dd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f0211dd0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f0211dd0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c0a9b40 con 0x7f020c0ae8f0 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0200011bd0 con 0x7f020c0ae8f0 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 1 --2- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 0x7f01f806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0200085980 con 0x7f020c0ae8f0 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 10 client.4794.objecter ms_dispatch 0x7f020c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 3 client.4794.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 3 client.4794.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:02.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f020affd700 1 --2- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 0x7f01f806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f020affd700 10 monclient: get_auth_request con 0x7f01f80688c0 auth_method 0 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.437+0000 7f01f37fe700 20 client.4794.objecter dump_active .. 0 homeless 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.438+0000 7f01f37fe700 10 client.4794.objecter ms_handle_connect 0x7f01f80688c0 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.438+0000 7f020affd700 1 --2- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 0x7f01f806ad70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f020c13fed0 tx=0x7f01fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.442+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0200053240 con 0x7f020c0ae8f0 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.442+0000 7f01f37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:02.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.442+0000 7f01f37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:02.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f0211dd0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:02.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f0211dd0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:02.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f0211dd0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f020c072960 con 0x7f020c0ae8f0 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f01f37fe700 1 -- 172.21.15.115:0/2532502583 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0200056860 con 0x7f020c0ae8f0 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f01f37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.663+0000 7f01f37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:02.664 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 10 librados: watch_flush enter 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 10 librados: watch_flush exit 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 20 client.4794.objecter shutdown clearing up homeless session... 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 10 client.4794.objecter successfully canceled tick 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 msgr2=0x7f01f806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 0x7f01f806ad70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f020c13fed0 tx=0x7f01fc009040 comp rx=0 tx=0).stop 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 10 monclient: shutdown 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:02.666 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 msgr2=0x7f020c13f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f020c0b0920 tx=0x7f0200017000 comp rx=0 tx=0).stop 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 shutdown_connections 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> v2:172.21.15.115:6816/115635 conn(0x7f01f80688c0 0x7f01f806ad70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f020c1415c0 0x7f020c0a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f020c0ae8f0 0x7f020c13f650 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 --2- 172.21.15.115:0/2532502583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c0aecc0 0x7f020c0a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 >> 172.21.15.115:0/2532502583 conn(0x7f020c06a460 msgr2=0x7f020c140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 shutdown_connections 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 -- 172.21.15.115:0/2532502583 wait complete. 2023-09-30T01:07:02.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:02.665+0000 7f0211dd0700 1 librados: shutdown 2023-09-30T01:07:05.677 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:05.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.805+0000 7f7e60a04700 10 monclient: get_monmap_and_config 2023-09-30T01:07:05.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.805+0000 7f7e60a04700 10 monclient: build_initial_monmap 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.805+0000 7f7e60a04700 10 monclient: monmap: 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:05.806491+0000 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:05.806491+0000 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:05.806 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:05.807 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:05.807 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:05.807 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:05.807 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 Processor -- start 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 -- start start 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient: init 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 0x7f7e5c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-c con 0x7f7e5c0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:05.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-a con 0x7f7e5c0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-b con 0x7f7e5c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a7040 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0b19b0 con 0x7f7e5c0ae8f0 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a57f0 con 0x7f7e5c1415c0 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.806+0000 7f7e60a04700 10 monclient(hunting): authenticate will time out at 1287.256931s 2023-09-30T01:07:05.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47054/0 (socket says 172.21.15.115:47054) 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50352/0 (socket says 172.21.15.115:50352) 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 -- 172.21.15.115:0/840689258 learned_addr learned my addr 172.21.15.115:0/840689258 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): get_auth_request con 0x7f7e5c1415c0 auth_method 0 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): get_auth_request con 0x7f7e5c0aecc0 auth_method 0 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:05.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 0x7f7e5c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 10 monclient(hunting): get_auth_request con 0x7f7e5c0ae8f0 auth_method 0 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_done global_id 4589 payload 274 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 -- 172.21.15.115:0/840689258 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 msgr2=0x7f7e5c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 0x7f7e5c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 -- 172.21.15.115:0/840689258 >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 msgr2=0x7f7e5c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 monclient: found mon.noname-b 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 -- 172.21.15.115:0/840689258 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0a5480 con 0x7f7e5c1415c0 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e59d9b700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:05.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 0x7f7e5c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5ad9d700 1 --2- 172.21.15.115:0/840689258 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0b59c0 tx=0x7f7e54009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc7c404110cee5f7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 1 -- 172.21.15.115:0/840689258 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5400e070 con 0x7f7e5c1415c0 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: dump: 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:05.821 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: _finish_auth 0 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 1 -- 172.21.15.115:0/840689258 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5400ba20 con 0x7f7e5c1415c0 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e60a04700 5 monclient: authenticate success, global_id 4589 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e60a04700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e60a04700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 1 -- 172.21.15.115:0/840689258 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e540092e0 con 0x7f7e5c1415c0 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e60a04700 4 set_mon_vals no callback set 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.807+0000 7f7e5959a700 10 monclient: dump: 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.823 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 10 monclient: shutdown 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 -- 172.21.15.115:0/840689258 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 msgr2=0x7f7e5c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 --2- 172.21.15.115:0/840689258 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0b59c0 tx=0x7f7e54009840 comp rx=0 tx=0).stop 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 -- 172.21.15.115:0/840689258 shutdown_connections 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 --2- 172.21.15.115:0/840689258 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c1415c0 0x7f7e5c0adb60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0ae8f0 0x7f7e5c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 --2- 172.21.15.115:0/840689258 >> v2:172.21.15.115:3300/0 conn(0x7f7e5c0aecc0 0x7f7e5c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 -- 172.21.15.115:0/840689258 >> 172.21.15.115:0/840689258 conn(0x7f7e5c06a460 msgr2=0x7f7e5c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 -- 172.21.15.115:0/840689258 shutdown_connections 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 1 -- 172.21.15.115:0/840689258 wait complete. 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 10 monclient: build_initial_monmap 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.808+0000 7f7e60a04700 10 monclient: monmap: 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:05.825 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 librados: starting msgr at 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 librados: starting objecter 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 Processor -- start 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 -- start start 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 librados: setting wanted keys 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 librados: calling monclient init 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient: init 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:07:05.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 0x7f7e5c13e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-b con 0x7f7e5c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-c con 0x7f7e5c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): picked mon.noname-a con 0x7f7e5c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c151830 con 0x7f7e5c1415c0 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0aba00 con 0x7f7e5c0ae8f0 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c151a90 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e60a04700 10 monclient(hunting): authenticate will time out at 1287.259871s 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e5ad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e59d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e5ad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50364/0 (socket says 172.21.15.115:50364) 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e5ad9d700 1 -- 172.21.15.115:0/2937233992 learned_addr learned my addr 172.21.15.115:0/2937233992 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e5ad9d700 10 monclient(hunting): get_auth_request con 0x7f7e5c1415c0 auth_method 0 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e59d9b700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47060/0 (socket says 172.21.15.115:47060) 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.809+0000 7f7e5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): get_auth_request con 0x7f7e5c0aecc0 auth_method 0 2023-09-30T01:07:05.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 0x7f7e5c13e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 -- 172.21.15.115:0/2937233992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 msgr2=0x7f7e5c13e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:05.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 0x7f7e5c13e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 -- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 msgr2=0x7f7e5c0abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 monclient: found mon.noname-c 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 -- 172.21.15.115:0/2937233992 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50009420 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5a59c700 10 monclient: get_auth_request con 0x7f7e5c0ae8f0 auth_method 0 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 0x7f7e5c13e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e5ad9d700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e59d9b700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ec80 tx=0x7f7e500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:05.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 client.?.objecter ms_handle_connect 0x7f7e5c0aecc0 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000c850 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: dump: 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.831 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5000c9b0 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 5 monclient: authenticate success, global_id 4592 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 10 monclient: _renew_subs 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e54009ae0 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:05.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000cd50 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e4a7fc700 10 monclient: dump: 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:05.833 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e58d99700 4 set_mon_vals no callback set 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 10 monclient: _renew_subs 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c0ac730 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.810+0000 7f7e60a04700 1 librados: init done 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.811+0000 7f7e60a04700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.811+0000 7f7e60a04700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.811+0000 7f7e60a04700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:05.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.811+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c072960 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.811+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7e50013680 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 1 --2- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 0x7f7e3c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7e50084190 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 10 client.4592.objecter ms_dispatch 0x7f7e5c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 3 client.4592.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 3 client.4592.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 0x7f7e3c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e5a59c700 10 monclient: get_auth_request con 0x7f7e3c068890 auth_method 0 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 20 client.4592.objecter dump_active .. 0 homeless 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e4a7fc700 10 client.4592.objecter ms_handle_connect 0x7f7e3c068890 2023-09-30T01:07:05.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.812+0000 7f7e5a59c700 1 --2- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 0x7f7e3c06ad40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006f00 tx=0x7f7e4c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:05.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.816+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e50051a50 con 0x7f7e5c0aecc0 2023-09-30T01:07:05.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.816+0000 7f7e4a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:05.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:05.816+0000 7f7e4a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:06.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.039+0000 7f7e60a04700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:06.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.039+0000 7f7e60a04700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:06.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.039+0000 7f7e60a04700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:06.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.039+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e5c0b2aa0 con 0x7f7e5c0aecc0 2023-09-30T01:07:06.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.040+0000 7f7e4a7fc700 1 -- 172.21.15.115:0/2937233992 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7e50055070 con 0x7f7e5c0aecc0 2023-09-30T01:07:06.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.040+0000 7f7e4a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:06.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.040+0000 7f7e4a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:06.041 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:06.042 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:06.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 10 librados: watch_flush enter 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 10 librados: watch_flush exit 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 20 client.4592.objecter shutdown clearing up homeless session... 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 10 client.4592.objecter successfully canceled tick 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 msgr2=0x7f7e3c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 0x7f7e3c06ad40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006f00 tx=0x7f7e4c008040 comp rx=0 tx=0).stop 2023-09-30T01:07:06.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 10 monclient: shutdown 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 msgr2=0x7f7e5c1512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ec80 tx=0x7f7e500091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 shutdown_connections 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> v2:172.21.15.115:6816/115635 conn(0x7f7e3c068890 0x7f7e3c06ad40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e5c1415c0 0x7f7e5c0abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0ae8f0 0x7f7e5c13e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 --2- 172.21.15.115:0/2937233992 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7e5c0aecc0 0x7f7e5c1512f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 >> 172.21.15.115:0/2937233992 conn(0x7f7e5c06a460 msgr2=0x7f7e5c144530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 shutdown_connections 2023-09-30T01:07:06.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 -- 172.21.15.115:0/2937233992 wait complete. 2023-09-30T01:07:06.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:06.042+0000 7f7e60a04700 1 librados: shutdown 2023-09-30T01:07:09.054 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:09.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.182+0000 7f6bc5049700 10 monclient: get_monmap_and_config 2023-09-30T01:07:09.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.182+0000 7f6bc5049700 10 monclient: build_initial_monmap 2023-09-30T01:07:09.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.182+0000 7f6bc5049700 10 monclient: monmap: 2023-09-30T01:07:09.183 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:09.183836+0000 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:09.183836+0000 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.183+0000 7f6bc5049700 1 Processor -- start 2023-09-30T01:07:09.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.183+0000 7f6bc5049700 1 -- start start 2023-09-30T01:07:09.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.183+0000 7f6bc5049700 10 monclient: init 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.183+0000 7f6bc5049700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bc00ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6bc00aecc0 0x7f6bc00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-a con 0x7f6bc00aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:09.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6bc01415c0 0x7f6bc00adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-c con 0x7f6bc01415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc00a7040 con 0x7f6bc00ae8f0 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc00a57f0 con 0x7f6bc00aecc0 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc00a5480 con 0x7f6bc01415c0 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient(hunting): authenticate will time out at 1290.634087s 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6bc00aecc0 0x7f6bc00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6bc00aecc0 0x7f6bc00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50378/0 (socket says 172.21.15.115:50378) 2023-09-30T01:07:09.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 -- 172.21.15.115:0/1248431331 learned_addr learned my addr 172.21.15.115:0/1248431331 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): get_auth_request con 0x7f6bc00aecc0 auth_method 0 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbf59e700 1 --2- 172.21.15.115:0/1248431331 >> v2:172.21.15.157:3300/0 conn(0x7f6bc01415c0 0x7f6bc00adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbf59e700 10 monclient(hunting): get_auth_request con 0x7f6bc01415c0 auth_method 0 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbf59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:09.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbf59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient(hunting): handle_auth_done global_id 4803 payload 274 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 -- 172.21.15.115:0/1248431331 >> v2:172.21.15.157:3300/0 conn(0x7f6bc01415c0 msgr2=0x7f6bc00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 --2- 172.21.15.115:0/1248431331 >> v2:172.21.15.157:3300/0 conn(0x7f6bc01415c0 0x7f6bc00adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 -- 172.21.15.115:0/1248431331 >> v1:172.21.15.157:6789/0 conn(0x7f6bc00ae8f0 legacy=0x7f6bc00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient: _finish_hunting 0 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 monclient: found mon.noname-a 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 -- 172.21.15.115:0/1248431331 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00a55c0 con 0x7f6bc00aecc0 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbe59c700 1 --2- 172.21.15.115:0/1248431331 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc00aecc0 0x7f6bc00a5d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00a6250 tx=0x7f6bb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e6ca501c3f16c4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 1 -- 172.21.15.115:0/1248431331 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000b860 con 0x7f6bc00aecc0 2023-09-30T01:07:09.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: dump: 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: _finish_auth 0 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 1 -- 172.21.15.115:0/1248431331 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000b9c0 con 0x7f6bc00aecc0 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 5 monclient: authenticate success, global_id 4803 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 1 -- 172.21.15.115:0/1248431331 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000bd60 con 0x7f6bc00aecc0 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:09.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bc5049700 4 set_mon_vals no callback set 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.184+0000 7f6bbdd9b700 10 monclient: dump: 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:09.201 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 10 monclient: shutdown 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 -- 172.21.15.115:0/1248431331 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc00aecc0 msgr2=0x7f6bc00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1248431331 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc00aecc0 0x7f6bc00a5d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00a6250 tx=0x7f6bb0009250 comp rx=0 tx=0).stop 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 -- 172.21.15.115:0/1248431331 shutdown_connections 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1248431331 >> v2:172.21.15.157:3300/0 conn(0x7f6bc01415c0 0x7f6bc00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1248431331 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc00aecc0 0x7f6bc00a5d10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 -- 172.21.15.115:0/1248431331 >> 172.21.15.115:0/1248431331 conn(0x7f6bc006a460 msgr2=0x7f6bc00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 -- 172.21.15.115:0/1248431331 shutdown_connections 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.185+0000 7f6bc5049700 1 -- 172.21.15.115:0/1248431331 wait complete. 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 10 monclient: build_initial_monmap 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 10 monclient: monmap: 2023-09-30T01:07:09.203 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 librados: starting msgr at 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 librados: starting objecter 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 Processor -- start 2023-09-30T01:07:09.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 -- start start 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 librados: setting wanted keys 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 1 librados: calling monclient init 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.186+0000 7f6bc5049700 10 monclient: init 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 0x7f6bc013e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-c con 0x7f6bc00ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bc00aecc0 0x7f6bc0151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-b con 0x7f6bc00aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): picked mon.noname-a con 0x7f6bc01415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:09.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00aa410 con 0x7f6bc01415c0 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00ab7b0 con 0x7f6bc00aecc0 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0151ca0 con 0x7f6bc00ae8f0 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient(hunting): authenticate will time out at 1290.637146s 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 0x7f6bc013e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:50386/0 (socket says 172.21.15.115:50386) 2023-09-30T01:07:09.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 0x7f6bc013e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47078/0 (socket says 172.21.15.115:47078) 2023-09-30T01:07:09.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 -- 172.21.15.115:0/1259062698 learned_addr learned my addr 172.21.15.115:0/1259062698 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): get_auth_request con 0x7f6bc01415c0 auth_method 0 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): get_auth_request con 0x7f6bc00ae8f0 auth_method 0 2023-09-30T01:07:09.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:09.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:09.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:09.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:09.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:09.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:09.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:09.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:09.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbe59c700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bc00aecc0 0x7f6bc0151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient(hunting): handle_auth_done global_id 4806 payload 824 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 -- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 msgr2=0x7f6bc013e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 0x7f6bc013e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 -- 172.21.15.115:0/1259062698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bc00aecc0 msgr2=0x7f6bc0151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bc00aecc0 0x7f6bc0151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient: _finish_hunting 0 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 monclient: found mon.noname-a 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 -- 172.21.15.115:0/1259062698 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00094f0 con 0x7f6bc01415c0 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbf59e700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007f50 tx=0x7f6bb8009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:09.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 client.?.objecter ms_handle_connect 0x7f6bc01415c0 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb800ec20 con 0x7f6bc01415c0 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: dump: 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.216 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 5 monclient: authenticate success, global_id 4806 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb800eda0 con 0x7f6bc01415c0 2023-09-30T01:07:09.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _renew_subs 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc00aa900 con 0x7f6bc01415c0 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8014670 con 0x7f6bc01415c0 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:09.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 client.4806.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6ba6ffd700 10 monclient: dump: 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:09.219 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _renew_subs 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bbd59a700 4 set_mon_vals no callback set 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc00aa110 con 0x7f6bc01415c0 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.187+0000 7f6bc5049700 1 librados: init done 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.188+0000 7f6bc5049700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.188+0000 7f6bc5049700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.188+0000 7f6bc5049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.188+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc00ab980 con 0x7f6bc01415c0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6bb80149e0 con 0x7f6bc01415c0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 1 --2- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 0x7f6bac06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6bbed9d700 1 --2- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 0x7f6bac06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6bb80845e0 con 0x7f6bc01415c0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 10 client.4806.objecter ms_dispatch 0x7f6bc0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 3 client.4806.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 3 client.4806.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6bbed9d700 10 monclient: get_auth_request con 0x7f6bac068890 auth_method 0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 20 client.4806.objecter dump_active .. 0 homeless 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6ba6ffd700 10 client.4806.objecter ms_handle_connect 0x7f6bac068890 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.189+0000 7f6bbed9d700 1 --2- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 0x7f6bac06ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4003a80 tx=0x7f6bb4007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:09.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.193+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bb8051ea0 con 0x7f6bc01415c0 2023-09-30T01:07:09.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.193+0000 7f6ba6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:09.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.193+0000 7f6ba6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:09.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6bc5049700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6bc5049700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6bc5049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bc0072960 con 0x7f6bc01415c0 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1259062698 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6bb80554c0 con 0x7f6bc01415c0 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6ba6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.417+0000 7f6ba6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:09.418 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:09.419 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:09.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.419+0000 7f6bc5049700 10 librados: watch_flush enter 2023-09-30T01:07:09.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 10 librados: watch_flush exit 2023-09-30T01:07:09.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 20 client.4806.objecter shutdown clearing up homeless session... 2023-09-30T01:07:09.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 10 client.4806.objecter successfully canceled tick 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 msgr2=0x7f6bac06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 0x7f6bac06ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4003a80 tx=0x7f6bb4007ec0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 10 monclient: shutdown 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 msgr2=0x7f6bc0151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007f50 tx=0x7f6bb8009860 comp rx=0 tx=0).stop 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 shutdown_connections 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> v2:172.21.15.115:6816/115635 conn(0x7f6bac068890 0x7f6bac06ad40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6bc01415c0 0x7f6bc0151760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6bc00ae8f0 0x7f6bc013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 --2- 172.21.15.115:0/1259062698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bc00aecc0 0x7f6bc0151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:09.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 >> 172.21.15.115:0/1259062698 conn(0x7f6bc006a460 msgr2=0x7f6bc00b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:09.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 shutdown_connections 2023-09-30T01:07:09.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 -- 172.21.15.115:0/1259062698 wait complete. 2023-09-30T01:07:09.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:09.420+0000 7f6bc5049700 1 librados: shutdown 2023-09-30T01:07:12.432 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.556+0000 7fb556c42700 10 monclient: get_monmap_and_config 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.556+0000 7fb556c42700 10 monclient: build_initial_monmap 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.556+0000 7fb556c42700 10 monclient: monmap: 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:12.557887+0000 2023-09-30T01:07:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:12.557887+0000 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:12.560 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.557+0000 7fb556c42700 1 Processor -- start 2023-09-30T01:07:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.557+0000 7fb556c42700 1 -- start start 2023-09-30T01:07:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.557+0000 7fb556c42700 10 monclient: init 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 0x7fb5500a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-c con 0x7fb5500ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-a con 0x7fb5500aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:12.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-b con 0x7fb5501415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500a7040 con 0x7fb5500aecc0 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500b19b0 con 0x7fb5500ae8f0 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500a57f0 con 0x7fb5501415c0 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb556c42700 10 monclient(hunting): authenticate will time out at 1294.008335s 2023-09-30T01:07:12.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:60476/0 (socket says 172.21.15.115:60476) 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33286/0 (socket says 172.21.15.115:33286) 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 -- 172.21.15.115:0/1158334711 learned_addr learned my addr 172.21.15.115:0/1158334711 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): get_auth_request con 0x7fb5501415c0 auth_method 0 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:12.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): get_auth_request con 0x7fb5500aecc0 auth_method 0 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:12.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5549de700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 0x7fb5500a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient(hunting): handle_auth_done global_id 4598 payload 274 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 -- 172.21.15.115:0/1158334711 >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 msgr2=0x7fb5500a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 0x7fb5500a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 -- 172.21.15.115:0/1158334711 >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 msgr2=0x7fb5500b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient: _finish_hunting 0 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 monclient: found mon.noname-b 2023-09-30T01:07:12.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 -- 172.21.15.115:0/1158334711 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500a5480 con 0x7fb5501415c0 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb54ffff700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5549de700 10 monclient: get_auth_request con 0x7fb5500ae8f0 auth_method 0 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5549de700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 0x7fb5500a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.558+0000 7fb5551df700 1 --2- 172.21.15.115:0/1158334711 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb5500b59c0 tx=0x7fb548009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bdec500ad35ea89 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 1 -- 172.21.15.115:0/1158334711 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54800e070 con 0x7fb5501415c0 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: dump: 2023-09-30T01:07:12.578 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:12.579 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:12.579 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.579 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.579 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:12.579 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 5 monclient: authenticate success, global_id 4598 2023-09-30T01:07:12.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 1 -- 172.21.15.115:0/1158334711 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54800ba20 con 0x7fb5501415c0 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 4 set_mon_vals no callback set 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 1 -- 172.21.15.115:0/1158334711 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480092e0 con 0x7fb5501415c0 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb54f7fe700 10 monclient: dump: 2023-09-30T01:07:12.581 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 10 monclient: shutdown 2023-09-30T01:07:12.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 -- 172.21.15.115:0/1158334711 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 msgr2=0x7fb5500adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 --2- 172.21.15.115:0/1158334711 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb5500b59c0 tx=0x7fb548009840 comp rx=0 tx=0).stop 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 -- 172.21.15.115:0/1158334711 shutdown_connections 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 --2- 172.21.15.115:0/1158334711 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5501415c0 0x7fb5500adb60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.157:3300/0 conn(0x7fb5500ae8f0 0x7fb5500a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 --2- 172.21.15.115:0/1158334711 >> v2:172.21.15.115:3300/0 conn(0x7fb5500aecc0 0x7fb5500b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 -- 172.21.15.115:0/1158334711 >> 172.21.15.115:0/1158334711 conn(0x7fb55006a460 msgr2=0x7fb5500630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 -- 172.21.15.115:0/1158334711 shutdown_connections 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.559+0000 7fb556c42700 1 -- 172.21.15.115:0/1158334711 wait complete. 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 10 monclient: build_initial_monmap 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 10 monclient: monmap: 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:12.584 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 librados: starting msgr at 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 librados: starting objecter 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 Processor -- start 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 -- start start 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 librados: setting wanted keys 2023-09-30T01:07:12.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 1 librados: calling monclient init 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.560+0000 7fb556c42700 10 monclient: init 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-a con 0x7fb5500ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 0x7fb55013e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-c con 0x7fb5500aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5501415c0 0x7fb550151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): picked mon.noname-b con 0x7fb5501415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500a87e0 con 0x7fb5500ae8f0 2023-09-30T01:07:12.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500a8950 con 0x7fb5501415c0 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500a9420 con 0x7fb5500aecc0 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb556c42700 10 monclient(hunting): authenticate will time out at 1294.011366s 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 0x7fb55013e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33300/0 (socket says 172.21.15.115:33300) 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 -- 172.21.15.115:0/532639808 learned_addr learned my addr 172.21.15.115:0/532639808 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): get_auth_request con 0x7fb5500ae8f0 auth_method 0 2023-09-30T01:07:12.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): get_auth_request con 0x7fb5500aecc0 auth_method 0 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:12.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 -- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 msgr2=0x7fb55013e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 0x7fb55013e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5551df700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5501415c0 0x7fb550151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 -- 172.21.15.115:0/532639808 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5501415c0 msgr2=0x7fb550151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5501415c0 0x7fb550151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient: _finish_hunting 0 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 monclient: found mon.noname-a 2023-09-30T01:07:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:12.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:12.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 -- 172.21.15.115:0/532639808 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c009420 con 0x7fb5500ae8f0 2023-09-30T01:07:12.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54ffff700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 0x7fb55013e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:12.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb5549de700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c860 tx=0x7fb544008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:12.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54cff9700 10 client.?.objecter ms_handle_connect 0x7fb5500ae8f0 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.561+0000 7fb54cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544011600 con 0x7fb5500ae8f0 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: dump: 2023-09-30T01:07:12.591 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:12.592 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: _finish_auth 0 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 5 monclient: authenticate success, global_id 4812 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _renew_subs 2023-09-30T01:07:12.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb544011760 con 0x7fb5500ae8f0 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb548009ae0 con 0x7fb5500ae8f0 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544011b20 con 0x7fb5500ae8f0 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:12.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54cff9700 10 monclient: dump: 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:12.595 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb54effd700 4 set_mon_vals no callback set 2023-09-30T01:07:12.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 client.4812.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _renew_subs 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5500aaab0 con 0x7fb5500ae8f0 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 1 librados: init done 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:12.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.562+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5500b0580 con 0x7fb5500ae8f0 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb5440193b0 con 0x7fb5500ae8f0 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 1 --2- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 0x7fb53406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb544083e60 con 0x7fb5500ae8f0 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54ffff700 1 --2- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 0x7fb53406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 10 client.4812.objecter ms_dispatch 0x7fb550001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 3 client.4812.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:12.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 3 client.4812.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54ffff700 10 monclient: get_auth_request con 0x7fb534068890 auth_method 0 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.563+0000 7fb54cff9700 20 client.4812.objecter dump_active .. 0 homeless 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.564+0000 7fb54cff9700 10 client.4812.objecter ms_handle_connect 0x7fb534068890 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.564+0000 7fb54ffff700 1 --2- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 0x7fb53406ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb5500aba20 tx=0x7fb53c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.567+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb544051720 con 0x7fb5500ae8f0 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.567+0000 7fb54cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:12.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.567+0000 7fb54cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.789+0000 7fb556c42700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.789+0000 7fb556c42700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.789+0000 7fb556c42700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.789+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb550072960 con 0x7fb5500ae8f0 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.790+0000 7fb54cff9700 1 -- 172.21.15.115:0/532639808 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb544015100 con 0x7fb5500ae8f0 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.790+0000 7fb54cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:12.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.790+0000 7fb54cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:12.791 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:12.791 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:12.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 10 librados: watch_flush enter 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 10 librados: watch_flush exit 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 20 client.4812.objecter shutdown clearing up homeless session... 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 10 client.4812.objecter successfully canceled tick 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 msgr2=0x7fb53406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 0x7fb53406ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb5500aba20 tx=0x7fb53c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 10 monclient: shutdown 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 msgr2=0x7fb5500a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:12.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c860 tx=0x7fb544008040 comp rx=0 tx=0).stop 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 shutdown_connections 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> v2:172.21.15.115:6816/115635 conn(0x7fb534068890 0x7fb53406ad40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5501415c0 0x7fb550151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5500ae8f0 0x7fb5500a9ce0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 --2- 172.21.15.115:0/532639808 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb5500aecc0 0x7fb55013e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 >> 172.21.15.115:0/532639808 conn(0x7fb55006a460 msgr2=0x7fb550144710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 shutdown_connections 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 -- 172.21.15.115:0/532639808 wait complete. 2023-09-30T01:07:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:12.792+0000 7fb556c42700 1 librados: shutdown 2023-09-30T01:07:15.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:15.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.933+0000 7feecea90700 10 monclient: get_monmap_and_config 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.933+0000 7feecea90700 10 monclient: build_initial_monmap 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.933+0000 7feecea90700 10 monclient: monmap: 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:15.934563+0000 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:15.934563+0000 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:15.934 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:15.935 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:15.935 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:15.935 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 Processor -- start 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 -- start start 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient: init 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7feec80ae8f0 0x7feec80a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): picked mon.noname-a con 0x7feec80ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feec80aecc0 0x7feec80b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): picked mon.noname-c con 0x7feec80aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feec81415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80a7040 con 0x7feec81415c0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec80b19b0 con 0x7feec80ae8f0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec80a57f0 con 0x7feec80aecc0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.934+0000 7feecea90700 10 monclient(hunting): authenticate will time out at 1297.384992s 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7feec80ae8f0 0x7feec80a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7feec80ae8f0 0x7feec80a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33310/0 (socket says 172.21.15.115:33310) 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecd02d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7feec81415c0 0x7feec80adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57264/0 (socket says 172.21.15.115:57264) 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 -- 172.21.15.115:0/2199288978 learned_addr learned my addr 172.21.15.115:0/2199288978 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): get_auth_request con 0x7feec80ae8f0 auth_method 0 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec7fff700 1 --2- 172.21.15.115:0/2199288978 >> v2:172.21.15.157:3300/0 conn(0x7feec80aecc0 0x7feec80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec7fff700 10 monclient(hunting): get_auth_request con 0x7feec80aecc0 auth_method 0 2023-09-30T01:07:15.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient(hunting): handle_auth_done global_id 4815 payload 274 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 -- 172.21.15.115:0/2199288978 >> v2:172.21.15.157:3300/0 conn(0x7feec80aecc0 msgr2=0x7feec80b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 --2- 172.21.15.115:0/2199288978 >> v2:172.21.15.157:3300/0 conn(0x7feec80aecc0 0x7feec80b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 -- 172.21.15.115:0/2199288978 >> v1:172.21.15.115:6789/0 conn(0x7feec81415c0 legacy=0x7feec80adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient: _finish_hunting 0 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 monclient: found mon.noname-a 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 -- 172.21.15.115:0/2199288978 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80a5480 con 0x7feec80ae8f0 2023-09-30T01:07:15.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 1 -- 172.21.15.115:0/2199288978 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745341550 0 0) 0x7feec80a7040 con 0x7feec81415c0 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecc82c700 1 --2- 172.21.15.115:0/2199288978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a5d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feec8064080 tx=0x7feebc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbed8b21a0e00fe9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 1 -- 172.21.15.115:0/2199288978 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00b860 con 0x7feec80ae8f0 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: dump: 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: _finish_auth 0 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 1 -- 172.21.15.115:0/2199288978 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc00b9c0 con 0x7feec80ae8f0 2023-09-30T01:07:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecea90700 5 monclient: authenticate success, global_id 4815 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecea90700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecea90700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 1 -- 172.21.15.115:0/2199288978 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00bd60 con 0x7feec80ae8f0 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feecea90700 4 set_mon_vals no callback set 2023-09-30T01:07:15.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.935+0000 7feec77fe700 10 monclient: dump: 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:15.953 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 10 monclient: shutdown 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 -- 172.21.15.115:0/2199288978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 msgr2=0x7feec80a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 --2- 172.21.15.115:0/2199288978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a5d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feec8064080 tx=0x7feebc009250 comp rx=0 tx=0).stop 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 -- 172.21.15.115:0/2199288978 shutdown_connections 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 --2- 172.21.15.115:0/2199288978 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a5d10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 --2- 172.21.15.115:0/2199288978 >> v2:172.21.15.157:3300/0 conn(0x7feec80aecc0 0x7feec80b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 -- 172.21.15.115:0/2199288978 >> 172.21.15.115:0/2199288978 conn(0x7feec806a460 msgr2=0x7feec80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 -- 172.21.15.115:0/2199288978 shutdown_connections 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 1 -- 172.21.15.115:0/2199288978 wait complete. 2023-09-30T01:07:15.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 10 monclient: build_initial_monmap 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.936+0000 7feecea90700 10 monclient: monmap: 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.955 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 librados: starting msgr at 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 librados: starting objecter 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 Processor -- start 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 -- start start 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 librados: setting wanted keys 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 1 librados: calling monclient init 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.937+0000 7feecea90700 10 monclient: init 2023-09-30T01:07:15.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): picked mon.noname-a con 0x7feec80ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): picked mon.noname-c con 0x7feec80aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec81415c0 0x7feec80a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): picked mon.noname-b con 0x7feec81415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec813e6c0 con 0x7feec80ae8f0 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80ab940 con 0x7feec81415c0 2023-09-30T01:07:15.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec80ad520 con 0x7feec80aecc0 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient(hunting): authenticate will time out at 1297.388095s 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:60482/0 (socket says 172.21.15.115:60482) 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 1 -- 172.21.15.115:0/3339035092 learned_addr learned my addr 172.21.15.115:0/3339035092 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33320/0 (socket says 172.21.15.115:33320) 2023-09-30T01:07:15.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): get_auth_request con 0x7feec80aecc0 auth_method 0 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): get_auth_request con 0x7feec80ae8f0 auth_method 0 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:15.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecd02d700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec81415c0 0x7feec80a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecd02d700 10 monclient(hunting): get_auth_request con 0x7feec81415c0 auth_method 0 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecd02d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecd02d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecd02d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient(hunting): handle_auth_done global_id 4821 payload 824 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 -- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 msgr2=0x7feec813e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 -- 172.21.15.115:0/3339035092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec81415c0 msgr2=0x7feec80a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:15.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec81415c0 0x7feec80a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient: _finish_hunting 0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 monclient: found mon.noname-a 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 -- 172.21.15.115:0/3339035092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0094f0 con 0x7feec80ae8f0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec7fff700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecc82c700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feec80a6280 tx=0x7feebc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 client.?.objecter ms_handle_connect 0x7feec80ae8f0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc024440 con 0x7feec80ae8f0 2023-09-30T01:07:15.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: dump: 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 5 monclient: authenticate success, global_id 4821 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc0245a0 con 0x7feec80ae8f0 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _renew_subs 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec80abd60 con 0x7feec80ae8f0 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc024940 con 0x7feec80ae8f0 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec4ff9700 10 monclient: dump: 2023-09-30T01:07:15.963 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feec6ffd700 4 set_mon_vals no callback set 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 client.4821.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _renew_subs 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec80ac050 con 0x7feec80ae8f0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.938+0000 7feecea90700 1 librados: init done 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.939+0000 7feecea90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.939+0000 7feecea90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.939+0000 7feecea90700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.939+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8072960 con 0x7feec80ae8f0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7feebc01a420 con 0x7feec80ae8f0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 1 --2- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 0x7feeac06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec7fff700 1 --2- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 0x7feeac06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7feebc085d00 con 0x7feec80ae8f0 2023-09-30T01:07:15.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 10 client.4821.objecter ms_dispatch 0x7feec8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 3 client.4821.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 3 client.4821.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec7fff700 10 monclient: get_auth_request con 0x7feeac068890 auth_method 0 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.940+0000 7feec4ff9700 20 client.4821.objecter dump_active .. 0 homeless 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.941+0000 7feec4ff9700 10 client.4821.objecter ms_handle_connect 0x7feeac068890 2023-09-30T01:07:15.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.941+0000 7feec7fff700 1 --2- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 0x7feeac06ad40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feec813eae0 tx=0x7feeb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:15.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.944+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feebc0535c0 con 0x7feec80ae8f0 2023-09-30T01:07:15.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.944+0000 7feec4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:15.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:15.944+0000 7feec4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.164+0000 7feecea90700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.164+0000 7feecea90700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.164+0000 7feecea90700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.164+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feec8141200 con 0x7feec80ae8f0 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.165+0000 7feec4ff9700 1 -- 172.21.15.115:0/3339035092 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7feebc015080 con 0x7feec80ae8f0 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.165+0000 7feec4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.165+0000 7feec4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:16.165 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:16.166 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:16.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.166+0000 7feecea90700 10 librados: watch_flush enter 2023-09-30T01:07:16.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.166+0000 7feecea90700 10 librados: watch_flush exit 2023-09-30T01:07:16.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.166+0000 7feecea90700 20 client.4821.objecter shutdown clearing up homeless session... 2023-09-30T01:07:16.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.166+0000 7feecea90700 10 client.4821.objecter successfully canceled tick 2023-09-30T01:07:16.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 msgr2=0x7feeac06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 0x7feeac06ad40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feec813eae0 tx=0x7feeb4008040 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 10 monclient: shutdown 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 msgr2=0x7feec80a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feec80a6280 tx=0x7feebc017000 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 shutdown_connections 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> v2:172.21.15.115:6816/115635 conn(0x7feeac068890 0x7feeac06ad40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec81415c0 0x7feec80a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feec80ae8f0 0x7feec80a94a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 --2- 172.21.15.115:0/3339035092 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7feec80aecc0 0x7feec813e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:16.168 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 >> 172.21.15.115:0/3339035092 conn(0x7feec806a460 msgr2=0x7feec80b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:16.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 shutdown_connections 2023-09-30T01:07:16.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 -- 172.21.15.115:0/3339035092 wait complete. 2023-09-30T01:07:16.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:16.167+0000 7feecea90700 1 librados: shutdown 2023-09-30T01:07:19.180 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:19.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.309+0000 7f8e2c7dd700 10 monclient: get_monmap_and_config 2023-09-30T01:07:19.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.309+0000 7f8e2c7dd700 10 monclient: build_initial_monmap 2023-09-30T01:07:19.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.309+0000 7f8e2c7dd700 10 monclient: monmap: 2023-09-30T01:07:19.312 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:19.312 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:19.312 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:19.310646+0000 2023-09-30T01:07:19.313 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:19.310646+0000 2023-09-30T01:07:19.313 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:19.313 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.314 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:19.314 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:19.314 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:19.315 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:19.315 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:19.315 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.310+0000 7f8e2c7dd700 1 Processor -- start 2023-09-30T01:07:19.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.310+0000 7f8e2c7dd700 1 -- start start 2023-09-30T01:07:19.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.310+0000 7f8e2c7dd700 10 monclient: init 2023-09-30T01:07:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T01:07:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 0x7f8e240a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-b con 0x7f8e240ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e240aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e241415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240a7040 con 0x7f8e240aecc0 2023-09-30T01:07:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240a57f0 con 0x7f8e241415c0 2023-09-30T01:07:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240a5480 con 0x7f8e240ae8f0 2023-09-30T01:07:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient(hunting): authenticate will time out at 1300.761158s 2023-09-30T01:07:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 0x7f8e240a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29d78700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f8e240aecc0 0x7f8e240b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57272/0 (socket says 172.21.15.115:57272) 2023-09-30T01:07:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): get_auth_request con 0x7f8e240ae8f0 auth_method 0 2023-09-30T01:07:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:19.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:19.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:19.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29d78700 1 -- 172.21.15.115:0/1762549696 learned_addr learned my addr 172.21.15.115:0/1762549696 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:19.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:19.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2023-09-30T01:07:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 -- 172.21.15.115:0/1762549696 >> v1:172.21.15.157:6789/0 conn(0x7f8e241415c0 legacy=0x7f8e240adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 1 -- 172.21.15.115:0/1762549696 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289775906 0 0) 0x7f8e240a7040 con 0x7f8e240aecc0 2023-09-30T01:07:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 -- 172.21.15.115:0/1762549696 >> v1:172.21.15.115:6789/0 conn(0x7f8e240aecc0 legacy=0x7f8e240b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient: _finish_hunting 0 2023-09-30T01:07:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 monclient: found mon.noname-b 2023-09-30T01:07:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 -- 172.21.15.115:0/1762549696 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e240a55c0 con 0x7f8e240ae8f0 2023-09-30T01:07:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2a579700 1 --2- 172.21.15.115:0/1762549696 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 0x7f8e240a5d10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009910 tx=0x7f8e200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee7ffd266fb7b5c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 1 -- 172.21.15.115:0/1762549696 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2000b820 con 0x7f8e240ae8f0 2023-09-30T01:07:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: dump: 2023-09-30T01:07:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:19.338 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: _finish_auth 0 2023-09-30T01:07:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 5 monclient: authenticate success, global_id 4607 2023-09-30T01:07:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:19.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 1 -- 172.21.15.115:0/1762549696 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2000b980 con 0x7f8e240ae8f0 2023-09-30T01:07:19.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:19.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 1 -- 172.21.15.115:0/1762549696 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2000bd20 con 0x7f8e240ae8f0 2023-09-30T01:07:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e2c7dd700 4 set_mon_vals no callback set 2023-09-30T01:07:19.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:19.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.311+0000 7f8e29577700 10 monclient: dump: 2023-09-30T01:07:19.342 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:19.342 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:19.342 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.343 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.343 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:19.343 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.343 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:19.343 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:19.344 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:19.344 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 10 monclient: shutdown 2023-09-30T01:07:19.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:19.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1762549696 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 msgr2=0x7f8e240a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1762549696 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 0x7f8e240a5d10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009910 tx=0x7f8e200091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1762549696 shutdown_connections 2023-09-30T01:07:19.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1762549696 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e240ae8f0 0x7f8e240a5d10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1762549696 >> 172.21.15.115:0/1762549696 conn(0x7f8e2406a460 msgr2=0x7f8e240630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:19.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1762549696 shutdown_connections 2023-09-30T01:07:19.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.312+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1762549696 wait complete. 2023-09-30T01:07:19.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient: build_initial_monmap 2023-09-30T01:07:19.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient: monmap: 2023-09-30T01:07:19.347 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:19.347 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:19.347 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:19.347 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:19.347 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:19.348 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.348 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:19.348 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:19.348 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:19.349 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 librados: starting msgr at 2023-09-30T01:07:19.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 librados: starting objecter 2023-09-30T01:07:19.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 Processor -- start 2023-09-30T01:07:19.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 -- start start 2023-09-30T01:07:19.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 librados: setting wanted keys 2023-09-30T01:07:19.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 librados: calling monclient init 2023-09-30T01:07:19.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient: init 2023-09-30T01:07:19.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:19.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:07:19.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e240ae8f0 0x7f8e2413d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:19.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-b con 0x7f8e240ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:19.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:19.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-a con 0x7f8e240aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:19.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 0x7f8e240a8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:19.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): picked mon.noname-c con 0x7f8e241415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:19.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:19.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240a8d30 con 0x7f8e240aecc0 2023-09-30T01:07:19.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:19.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240a88d0 con 0x7f8e240ae8f0 2023-09-30T01:07:19.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:19.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240a8a40 con 0x7f8e241415c0 2023-09-30T01:07:19.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:19.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.313+0000 7f8e2c7dd700 10 monclient(hunting): authenticate will time out at 1300.763965s 2023-09-30T01:07:19.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:19.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 0x7f8e240a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:19.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33322/0 (socket says 172.21.15.115:33322) 2023-09-30T01:07:19.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 -- 172.21.15.115:0/1425976183 learned_addr learned my addr 172.21.15.115:0/1425976183 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:19.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): get_auth_request con 0x7f8e240aecc0 auth_method 0 2023-09-30T01:07:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:19.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): get_auth_request con 0x7f8e241415c0 auth_method 0 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:19.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2a579700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e240ae8f0 0x7f8e2413d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2a579700 10 monclient(hunting): get_auth_request con 0x7f8e240ae8f0 auth_method 0 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2a579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2a579700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2a579700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient(hunting): handle_auth_done global_id 4827 payload 824 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 -- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 msgr2=0x7f8e240a8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 0x7f8e240a8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 -- 172.21.15.115:0/1425976183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e240ae8f0 msgr2=0x7f8e2413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e240ae8f0 0x7f8e2413d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient: _finish_hunting 0 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 monclient: found mon.noname-a 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 -- 172.21.15.115:0/1425976183 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20009420 con 0x7f8e240aecc0 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2ad7a700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 0x7f8e240a8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:19.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e29d78700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c7e0 tx=0x7f8e14007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 client.?.objecter ms_handle_connect 0x7f8e240aecc0 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14005b30 con 0x7f8e240aecc0 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: dump: 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.360 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:19.361 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.361 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:19.361 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:19.361 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:19.361 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 5 monclient: authenticate success, global_id 4827 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14005c90 con 0x7f8e240aecc0 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 10 monclient: _renew_subs 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e24151ce0 con 0x7f8e240aecc0 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14007300 con 0x7f8e240aecc0 2023-09-30T01:07:19.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e1a7fc700 10 monclient: dump: 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e28d76700 4 set_mon_vals no callback set 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 10 client.4827.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 10 monclient: _renew_subs 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e240a9380 con 0x7f8e240aecc0 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.314+0000 7f8e2c7dd700 1 librados: init done 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.315+0000 7f8e2c7dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.315+0000 7f8e2c7dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.315+0000 7f8e2c7dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.315+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e240a8ee0 con 0x7f8e240aecc0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8e140076b0 con 0x7f8e240aecc0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 1 --2- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 0x7f8e0806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8e140835e0 con 0x7f8e240aecc0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 10 client.4827.objecter ms_dispatch 0x7f8e24001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 3 client.4827.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 3 client.4827.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e2a579700 1 --2- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 0x7f8e0806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e2a579700 10 monclient: get_auth_request con 0x7f8e08068890 auth_method 0 2023-09-30T01:07:19.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.316+0000 7f8e1a7fc700 20 client.4827.objecter dump_active .. 0 homeless 2023-09-30T01:07:19.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.317+0000 7f8e1a7fc700 10 client.4827.objecter ms_handle_connect 0x7f8e08068890 2023-09-30T01:07:19.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.317+0000 7f8e2a579700 1 --2- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 0x7f8e0806ad40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e200030c0 tx=0x7f8e20016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:19.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.320+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e14050f20 con 0x7f8e240aecc0 2023-09-30T01:07:19.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.320+0000 7f8e1a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:19.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.320+0000 7f8e1a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.542+0000 7f8e2c7dd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.542+0000 7f8e2c7dd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.542+0000 7f8e2c7dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.542+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8e240a8ee0 con 0x7f8e240aecc0 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.543+0000 7f8e1a7fc700 1 -- 172.21.15.115:0/1425976183 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8e14017100 con 0x7f8e240aecc0 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.543+0000 7f8e1a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:19.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.543+0000 7f8e1a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:19.544 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:19.544 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.544+0000 7f8e2c7dd700 10 librados: watch_flush enter 2023-09-30T01:07:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.544+0000 7f8e2c7dd700 10 librados: watch_flush exit 2023-09-30T01:07:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.544+0000 7f8e2c7dd700 20 client.4827.objecter shutdown clearing up homeless session... 2023-09-30T01:07:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.544+0000 7f8e2c7dd700 10 client.4827.objecter successfully canceled tick 2023-09-30T01:07:19.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 msgr2=0x7f8e0806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 0x7f8e0806ad40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e200030c0 tx=0x7f8e20016000 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 10 monclient: shutdown 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 msgr2=0x7f8e24151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c7e0 tx=0x7f8e14007df0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 shutdown_connections 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> v2:172.21.15.115:6816/115635 conn(0x7f8e08068890 0x7f8e0806ad40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8e241415c0 0x7f8e240a8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e240ae8f0 0x7f8e2413d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 --2- 172.21.15.115:0/1425976183 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8e240aecc0 0x7f8e24151200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:19.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 >> 172.21.15.115:0/1425976183 conn(0x7f8e2406a460 msgr2=0x7f8e2413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:19.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 shutdown_connections 2023-09-30T01:07:19.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 -- 172.21.15.115:0/1425976183 wait complete. 2023-09-30T01:07:19.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:19.545+0000 7f8e2c7dd700 1 librados: shutdown 2023-09-30T01:07:22.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.686+0000 7f76a7720700 10 monclient: get_monmap_and_config 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.686+0000 7f76a7720700 10 monclient: build_initial_monmap 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.686+0000 7f76a7720700 10 monclient: monmap: 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:22.687 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:22.687385+0000 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:22.687385+0000 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.686+0000 7f76a7720700 1 Processor -- start 2023-09-30T01:07:22.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 1 -- start start 2023-09-30T01:07:22.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient: init 2023-09-30T01:07:22.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f76a00ae8f0 0x7f76a00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-a con 0x7f76a00ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76a00aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76a01415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:22.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a00a7040 con 0x7f76a01415c0 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a00a57f0 con 0x7f76a00aecc0 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a00a5480 con 0x7f76a00ae8f0 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a7720700 10 monclient(hunting): authenticate will time out at 1304.137838s 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a54bc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f76a00ae8f0 0x7f76a00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a5cbd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f76a01415c0 0x7f76a00adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:55158/0 (socket says 172.21.15.115:55158) 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a54bc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f76a00ae8f0 0x7f76a00a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:41188/0 (socket says 172.21.15.115:41188) 2023-09-30T01:07:22.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.687+0000 7f76a5cbd700 1 -- 172.21.15.115:0/3290418344 learned_addr learned my addr 172.21.15.115:0/3290418344 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): get_auth_request con 0x7f76a00ae8f0 auth_method 0 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient(hunting): handle_auth_done global_id 4830 payload 274 2023-09-30T01:07:22.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 1 -- 172.21.15.115:0/3290418344 >> v1:172.21.15.157:6789/0 conn(0x7f76a00aecc0 legacy=0x7f76a00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 1 -- 172.21.15.115:0/3290418344 >> v1:172.21.15.115:6789/0 conn(0x7f76a01415c0 legacy=0x7f76a00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient: _finish_hunting 0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 1 monclient: found mon.noname-a 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 1 -- 172.21.15.115:0/3290418344 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a00a55c0 con 0x7f76a00ae8f0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 1 -- 172.21.15.115:0/3290418344 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935806348 0 0) 0x7f76a00a7040 con 0x7f76a01415c0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a54bc700 1 --2- 172.21.15.115:0/3290418344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a00ae8f0 0x7f76a00a5d10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76900098e0 tx=0x7f76900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d794c1bcf0272541 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 1 -- 172.21.15.115:0/3290418344 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690009e80 con 0x7f76a00ae8f0 2023-09-30T01:07:22.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: dump: 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: _finish_auth 0 2023-09-30T01:07:22.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a7720700 5 monclient: authenticate success, global_id 4830 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a7720700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 1 -- 172.21.15.115:0/3290418344 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769000b860 con 0x7f76a00ae8f0 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a7720700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f76a7720700 4 set_mon_vals no callback set 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 1 -- 172.21.15.115:0/3290418344 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769000bbe0 con 0x7f76a00ae8f0 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:22.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:22.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.688+0000 7f7697fff700 10 monclient: dump: 2023-09-30T01:07:22.707 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:22.707 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:22.707 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.707 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 10 monclient: shutdown 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 -- 172.21.15.115:0/3290418344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a00ae8f0 msgr2=0x7f76a00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 --2- 172.21.15.115:0/3290418344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a00ae8f0 0x7f76a00a5d10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76900098e0 tx=0x7f76900091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 -- 172.21.15.115:0/3290418344 shutdown_connections 2023-09-30T01:07:22.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 --2- 172.21.15.115:0/3290418344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a00ae8f0 0x7f76a00a5d10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 -- 172.21.15.115:0/3290418344 >> 172.21.15.115:0/3290418344 conn(0x7f76a006a460 msgr2=0x7f76a00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 -- 172.21.15.115:0/3290418344 shutdown_connections 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 1 -- 172.21.15.115:0/3290418344 wait complete. 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 10 monclient: build_initial_monmap 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.689+0000 7f76a7720700 10 monclient: monmap: 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.709 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 librados: starting msgr at 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 librados: starting objecter 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 Processor -- start 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 -- start start 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 librados: setting wanted keys 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 librados: calling monclient init 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient: init 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:22.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76a00ae8f0 0x7f76a00aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-b con 0x7f76a00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f76a00aecc0 0x7f76a013e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-c con 0x7f76a00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): picked mon.noname-a con 0x7f76a01415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00a9e20 con 0x7f76a01415c0 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00ab4b0 con 0x7f76a00ae8f0 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00ab720 con 0x7f76a00aecc0 2023-09-30T01:07:22.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.690+0000 7f76a7720700 10 monclient(hunting): authenticate will time out at 1304.140999s 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:41190/0 (socket says 172.21.15.115:41190) 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 -- 172.21.15.115:0/3522920829 learned_addr learned my addr 172.21.15.115:0/3522920829 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): get_auth_request con 0x7f76a01415c0 auth_method 0 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f76a00aecc0 0x7f76a013e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): get_auth_request con 0x7f76a00aecc0 auth_method 0 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:22.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76a00ae8f0 0x7f76a00aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): get_auth_request con 0x7f76a00ae8f0 auth_method 0 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:22.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a4cbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a54bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient(hunting): handle_auth_done global_id 4836 payload 824 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 -- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f76a00aecc0 msgr2=0x7f76a013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f76a00aecc0 0x7f76a013e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 -- 172.21.15.115:0/3522920829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76a00ae8f0 msgr2=0x7f76a00aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76a00ae8f0 0x7f76a00aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient: _finish_hunting 0 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 monclient: found mon.noname-a 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:22.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 -- 172.21.15.115:0/3522920829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7690009420 con 0x7f76a01415c0 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a5cbd700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f769800ccc0 tx=0x7f7698001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 client.?.objecter ms_handle_connect 0x7f76a01415c0 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800a8a0 con 0x7f76a01415c0 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: dump: 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.715 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: _finish_auth 0 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769800aa00 con 0x7f76a01415c0 2023-09-30T01:07:22.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 5 monclient: authenticate success, global_id 4836 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 10 monclient: _renew_subs 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a013ee40 con 0x7f76a01415c0 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800adc0 con 0x7f76a01415c0 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76957fa700 10 monclient: dump: 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:22.717 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76977fe700 4 set_mon_vals no callback set 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 10 client.4836.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 10 monclient: _renew_subs 2023-09-30T01:07:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a00a97c0 con 0x7f76a01415c0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.691+0000 7f76a7720700 1 librados: init done 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.692+0000 7f76a7720700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.692+0000 7f76a7720700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.692+0000 7f76a7720700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.692+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a00a9fd0 con 0x7f76a01415c0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7698007300 con 0x7f76a01415c0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 1 --2- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 0x7f768406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7698084160 con 0x7f76a01415c0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 10 client.4836.objecter ms_dispatch 0x7f76a0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 3 client.4836.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 3 client.4836.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:22.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76a54bc700 1 --2- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 0x7f768406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76a54bc700 10 monclient: get_auth_request con 0x7f7684068890 auth_method 0 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 20 client.4836.objecter dump_active .. 0 homeless 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76957fa700 10 client.4836.objecter ms_handle_connect 0x7f7684068890 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.693+0000 7f76a54bc700 1 --2- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 0x7f768406ad40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76900093f0 tx=0x7f7690017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.697+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76a00a9fd0 con 0x7f76a01415c0 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.697+0000 7f76957fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:22.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.697+0000 7f76957fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:22.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76a7720700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:22.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76a7720700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76a7720700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f76a00a9fd0 con 0x7f76a01415c0 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76957fa700 1 -- 172.21.15.115:0/3522920829 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f76a00a9fd0 con 0x7f76a01415c0 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76957fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.921+0000 7f76957fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:22.922 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 10 librados: watch_flush enter 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 10 librados: watch_flush exit 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 20 client.4836.objecter shutdown clearing up homeless session... 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 10 client.4836.objecter successfully canceled tick 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 msgr2=0x7f768406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 0x7f768406ad40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76900093f0 tx=0x7f7690017000 comp rx=0 tx=0).stop 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 10 monclient: shutdown 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 msgr2=0x7f76a0151000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f769800ccc0 tx=0x7f7698001a80 comp rx=0 tx=0).stop 2023-09-30T01:07:22.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 shutdown_connections 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> v2:172.21.15.115:6816/115635 conn(0x7f7684068890 0x7f768406ad40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76a01415c0 0x7f76a0151000 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76a00ae8f0 0x7f76a00aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 --2- 172.21.15.115:0/3522920829 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f76a00aecc0 0x7f76a013e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 >> 172.21.15.115:0/3522920829 conn(0x7f76a006a460 msgr2=0x7f76a0144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 shutdown_connections 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 -- 172.21.15.115:0/3522920829 wait complete. 2023-09-30T01:07:22.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:22.923+0000 7f76a7720700 1 librados: shutdown 2023-09-30T01:07:25.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:26.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.064+0000 7f2b40d7c700 10 monclient: get_monmap_and_config 2023-09-30T01:07:26.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.064+0000 7f2b40d7c700 10 monclient: build_initial_monmap 2023-09-30T01:07:26.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.064+0000 7f2b40d7c700 10 monclient: monmap: 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:26.065537+0000 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:26.065537+0000 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:26.065 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 Processor -- start 2023-09-30T01:07:26.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 -- start start 2023-09-30T01:07:26.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient: init 2023-09-30T01:07:26.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:26.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-a con 0x7f2b3c0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b3c0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2b3c1415c0 0x7f2b3c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-c con 0x7f2b3c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0a7040 con 0x7f2b3c0aecc0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0a57f0 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0a5480 con 0x7f2b3c1415c0 2023-09-30T01:07:26.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.065+0000 7f2b40d7c700 10 monclient(hunting): authenticate will time out at 1307.515947s 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:41206/0 (socket says 172.21.15.115:41206) 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 -- 172.21.15.115:0/4035825145 learned_addr learned my addr 172.21.15.115:0/4035825145 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): get_auth_request con 0x7f2b3c0ae8f0 auth_method 0 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:26.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3ad9d700 1 --2- 172.21.15.115:0/4035825145 >> v2:172.21.15.157:3300/0 conn(0x7f2b3c1415c0 0x7f2b3c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3ad9d700 10 monclient(hunting): get_auth_request con 0x7f2b3c1415c0 auth_method 0 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_done global_id 4839 payload 274 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 -- 172.21.15.115:0/4035825145 >> v2:172.21.15.157:3300/0 conn(0x7f2b3c1415c0 msgr2=0x7f2b3c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 --2- 172.21.15.115:0/4035825145 >> v2:172.21.15.157:3300/0 conn(0x7f2b3c1415c0 0x7f2b3c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 -- 172.21.15.115:0/4035825145 >> v1:172.21.15.157:6789/0 conn(0x7f2b3c0aecc0 legacy=0x7f2b3c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 monclient: found mon.noname-a 2023-09-30T01:07:26.077 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 -- 172.21.15.115:0/4035825145 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0a55c0 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3a59c700 1 --2- 172.21.15.115:0/4035825145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a6250 tx=0x7f2b2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35295ea789a1099 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 1 -- 172.21.15.115:0/4035825145 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c00b860 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: dump: 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.078 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: _finish_auth 0 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 1 -- 172.21.15.115:0/4035825145 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c00b9c0 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b40d7c700 5 monclient: authenticate success, global_id 4839 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b40d7c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:26.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b40d7c700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b40d7c700 4 set_mon_vals no callback set 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 1 -- 172.21.15.115:0/4035825145 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c00bd60 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.066+0000 7f2b3959a700 10 monclient: dump: 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:26.080 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 10 monclient: shutdown 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 -- 172.21.15.115:0/4035825145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c0ae8f0 msgr2=0x7f2b3c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/4035825145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a6250 tx=0x7f2b2c009250 comp rx=0 tx=0).stop 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 -- 172.21.15.115:0/4035825145 shutdown_connections 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/4035825145 >> v2:172.21.15.157:3300/0 conn(0x7f2b3c1415c0 0x7f2b3c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/4035825145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c0a5d10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 -- 172.21.15.115:0/4035825145 >> 172.21.15.115:0/4035825145 conn(0x7f2b3c06a460 msgr2=0x7f2b3c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:26.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 -- 172.21.15.115:0/4035825145 shutdown_connections 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 1 -- 172.21.15.115:0/4035825145 wait complete. 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 10 monclient: build_initial_monmap 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.067+0000 7f2b40d7c700 10 monclient: monmap: 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:26.082 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 librados: starting msgr at 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 librados: starting objecter 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 Processor -- start 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 -- start start 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 librados: setting wanted keys 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 librados: calling monclient init 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient: init 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c13ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-b con 0x7f2b3c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 0x7f2b3c0aa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-c con 0x7f2b3c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): picked mon.noname-a con 0x7f2b3c1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0aa9d0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0a8d30 con 0x7f2b3c0ae8f0 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0aa640 con 0x7f2b3c0aecc0 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:26.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b40d7c700 10 monclient(hunting): authenticate will time out at 1307.518817s 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b39d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 0x7f2b3c0aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:41208/0 (socket says 172.21.15.115:41208) 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b39d9b700 10 monclient(hunting): get_auth_request con 0x7f2b3c0aecc0 auth_method 0 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 1 -- 172.21.15.115:0/1977524358 learned_addr learned my addr 172.21.15.115:0/1977524358 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b39d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b39d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b39d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 10 monclient(hunting): get_auth_request con 0x7f2b3c1415c0 auth_method 0 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.068+0000 7f2b3a59c700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c13ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): get_auth_request con 0x7f2b3c0ae8f0 auth_method 0 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b39d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b39d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b39d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient(hunting): handle_auth_done global_id 4842 payload 824 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 -- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 msgr2=0x7f2b3c0aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 0x7f2b3c0aa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 -- 172.21.15.115:0/1977524358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b3c0ae8f0 msgr2=0x7f2b3c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c13ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 monclient: found mon.noname-a 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 -- 172.21.15.115:0/1977524358 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c0094f0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b39d9b700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 0x7f2b3c0aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b3ad9d700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007f20 tx=0x7f2b34009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 client.?.objecter ms_handle_connect 0x7f2b3c1415c0 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400ec20 con 0x7f2b3c1415c0 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: dump: 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 5 monclient: authenticate success, global_id 4842 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3400eda0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 10 monclient: _renew_subs 2023-09-30T01:07:26.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c13a110 con 0x7f2b3c1415c0 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34014670 con 0x7f2b3c1415c0 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b2a7fc700 10 monclient: dump: 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b38d99700 4 set_mon_vals no callback set 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 10 client.4842.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 10 monclient: _renew_subs 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c13a650 con 0x7f2b3c1415c0 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.069+0000 7f2b40d7c700 1 librados: init done 2023-09-30T01:07:26.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.070+0000 7f2b40d7c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.070+0000 7f2b40d7c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.070+0000 7f2b40d7c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.070+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c072960 con 0x7f2b3c1415c0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.070+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2b340149e0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 1 --2- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 0x7f2b1c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2b34084500 con 0x7f2b3c1415c0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b3a59c700 1 --2- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 0x7f2b1c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 10 client.4842.objecter ms_dispatch 0x7f2b3c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 3 client.4842.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 3 client.4842.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b3a59c700 10 monclient: get_auth_request con 0x7f2b1c068890 auth_method 0 2023-09-30T01:07:26.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 20 client.4842.objecter dump_active .. 0 homeless 2023-09-30T01:07:26.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b2a7fc700 10 client.4842.objecter ms_handle_connect 0x7f2b1c068890 2023-09-30T01:07:26.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.071+0000 7f2b3a59c700 1 --2- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 0x7f2b1c06ad40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a6250 tx=0x7f2b2c017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:26.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.075+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b34051dc0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.075+0000 7f2b2a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:26.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.075+0000 7f2b2a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.297+0000 7f2b40d7c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.297+0000 7f2b40d7c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.297+0000 7f2b40d7c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.297+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2b3c141a40 con 0x7f2b3c1415c0 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.298+0000 7f2b2a7fc700 1 -- 172.21.15.115:0/1977524358 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2b340553e0 con 0x7f2b3c1415c0 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.298+0000 7f2b2a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.298+0000 7f2b2a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:26.298 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:26.299 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.299+0000 7f2b40d7c700 10 librados: watch_flush enter 2023-09-30T01:07:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.299+0000 7f2b40d7c700 10 librados: watch_flush exit 2023-09-30T01:07:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.299+0000 7f2b40d7c700 20 client.4842.objecter shutdown clearing up homeless session... 2023-09-30T01:07:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.299+0000 7f2b40d7c700 10 client.4842.objecter successfully canceled tick 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 msgr2=0x7f2b1c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 0x7f2b1c06ad40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a6250 tx=0x7f2b2c017000 comp rx=0 tx=0).stop 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 10 monclient: shutdown 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 msgr2=0x7f2b3c139bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007f20 tx=0x7f2b34009860 comp rx=0 tx=0).stop 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 shutdown_connections 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> v2:172.21.15.115:6816/115635 conn(0x7f2b1c068890 0x7f2b1c06ad40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b3c1415c0 0x7f2b3c139bd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b3c0ae8f0 0x7f2b3c13ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 --2- 172.21.15.115:0/1977524358 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2b3c0aecc0 0x7f2b3c0aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 >> 172.21.15.115:0/1977524358 conn(0x7f2b3c06a460 msgr2=0x7f2b3c0b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 shutdown_connections 2023-09-30T01:07:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 -- 172.21.15.115:0/1977524358 wait complete. 2023-09-30T01:07:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:26.300+0000 7f2b40d7c700 1 librados: shutdown 2023-09-30T01:07:29.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:29.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.442+0000 7f105c77b700 10 monclient: get_monmap_and_config 2023-09-30T01:07:29.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.442+0000 7f105c77b700 10 monclient: build_initial_monmap 2023-09-30T01:07:29.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.442+0000 7f105c77b700 10 monclient: monmap: 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:29.443475+0000 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:29.443475+0000 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:29.443 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 1 Processor -- start 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 1 -- start start 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient: init 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T01:07:29.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 0x7f10541415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-a con 0x7f10540ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 0x7f10540b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-b con 0x7f10540aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10540a5c80 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.443+0000 7f105c77b700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540a7040 con 0x7f10540a5c80 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10540a5480 con 0x7f10540ae8f0 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540a5760 con 0x7f10540aecc0 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:29.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 10 monclient(hunting): authenticate will time out at 1310.894016s 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 0x7f10541415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 0x7f10540b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105ad18700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f10540a5c80 0x7f10540adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:55162/0 (socket says 172.21.15.115:55162) 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105ad18700 1 -- 172.21.15.115:0/917523326 learned_addr learned my addr 172.21.15.115:0/917523326 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): get_auth_request con 0x7f10540ae8f0 auth_method 0 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): get_auth_request con 0x7f10540aecc0 auth_method 0 2023-09-30T01:07:29.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient(hunting): handle_auth_done global_id 4619 payload 274 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 -- 172.21.15.115:0/917523326 >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 msgr2=0x7f10541415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 --2- 172.21.15.115:0/917523326 >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 0x7f10541415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 -- 172.21.15.115:0/917523326 >> v1:172.21.15.115:6789/0 conn(0x7f10540a5c80 legacy=0x7f10540adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient: _finish_hunting 0 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 monclient: found mon.noname-b 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 1 -- 172.21.15.115:0/917523326 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674031715 0 0) 0x7f10540a7040 con 0x7f10540a5c80 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 -- 172.21.15.115:0/917523326 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1054141b00 con 0x7f10540aecc0 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105a517700 1 --2- 172.21.15.115:0/917523326 >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 0x7f10541415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059d16700 1 --2- 172.21.15.115:0/917523326 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 0x7f10540b1430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f10440098e0 tx=0x7f10440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d56b255188410f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 1 -- 172.21.15.115:0/917523326 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104400b640 con 0x7f10540aecc0 2023-09-30T01:07:29.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: dump: 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: _finish_auth 0 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 5 monclient: authenticate success, global_id 4619 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 1 -- 172.21.15.115:0/917523326 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104400b7a0 con 0x7f10540aecc0 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f105c77b700 4 set_mon_vals no callback set 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 1 -- 172.21.15.115:0/917523326 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104400bb40 con 0x7f10540aecc0 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:29.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.444+0000 7f1059515700 10 monclient: dump: 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 10 monclient: shutdown 2023-09-30T01:07:29.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 -- 172.21.15.115:0/917523326 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 msgr2=0x7f10540b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 --2- 172.21.15.115:0/917523326 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 0x7f10540b1430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f10440098e0 tx=0x7f10440091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 -- 172.21.15.115:0/917523326 shutdown_connections 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 --2- 172.21.15.115:0/917523326 >> v2:172.21.15.115:3300/0 conn(0x7f10540ae8f0 0x7f10541415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 --2- 172.21.15.115:0/917523326 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540aecc0 0x7f10540b1430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 -- 172.21.15.115:0/917523326 >> 172.21.15.115:0/917523326 conn(0x7f105406a490 msgr2=0x7f10540630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 -- 172.21.15.115:0/917523326 shutdown_connections 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.445+0000 7f105c77b700 1 -- 172.21.15.115:0/917523326 wait complete. 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 10 monclient: build_initial_monmap 2023-09-30T01:07:29.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 10 monclient: monmap: 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:29.461 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 librados: starting msgr at 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 librados: starting objecter 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 Processor -- start 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 -- start start 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 librados: setting wanted keys 2023-09-30T01:07:29.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 1 librados: calling monclient init 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.446+0000 7f105c77b700 10 monclient: init 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-c con 0x7f10540a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 0x7f105413d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-b con 0x7f10540ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 0x7f105413d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): picked mon.noname-a con 0x7f10540aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540a9860 con 0x7f10540aecc0 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540aa2c0 con 0x7f10540ae8f0 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540aa5a0 con 0x7f10540a5c80 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105c77b700 10 monclient(hunting): authenticate will time out at 1310.897675s 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 0x7f105413d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:49806/0 (socket says 172.21.15.115:49806) 2023-09-30T01:07:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 1 -- 172.21.15.115:0/1734383571 learned_addr learned my addr 172.21.15.115:0/1734383571 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): get_auth_request con 0x7f10540a5c80 auth_method 0 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): get_auth_request con 0x7f10540aecc0 auth_method 0 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105ad18700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.447+0000 7f105a517700 10 monclient(hunting): handle_auth_done global_id 4622 payload 824 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f1059d16700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 0x7f105413d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 -- 172.21.15.115:0/1734383571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 msgr2=0x7f105413d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 0x7f105413d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 -- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 msgr2=0x7f105413d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 0x7f105413d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 10 monclient: _finish_hunting 0 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 monclient: found mon.noname-c 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:29.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 -- 172.21.15.115:0/1734383571 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050009420 con 0x7f10540a5c80 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f1059d16700 10 monclient: get_auth_request con 0x7f10540ae8f0 auth_method 0 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105ad18700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 0x7f105413d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f1059d16700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 0x7f105413d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105a517700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f105000ec80 tx=0x7f10500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 client.?.objecter ms_handle_connect 0x7f10540a5c80 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000c8f0 con 0x7f10540a5c80 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: dump: 2023-09-30T01:07:29.471 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:29.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 5 monclient: authenticate success, global_id 4622 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105000ca50 con 0x7f10540a5c80 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105000cdd0 con 0x7f10540a5c80 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 10 monclient: _renew_subs 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1044009420 con 0x7f10540a5c80 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f1058d14700 4 set_mon_vals no callback set 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f104a7fc700 10 monclient: dump: 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 10 monclient: _renew_subs 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10540aa930 con 0x7f10540a5c80 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.448+0000 7f105c77b700 1 librados: init done 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.449+0000 7f105c77b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.449+0000 7f105c77b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.449+0000 7f105c77b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.449+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1054141dd0 con 0x7f10540a5c80 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.449+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1050013680 con 0x7f10540a5c80 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 1 --2- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 0x7f103806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1050084270 con 0x7f10540a5c80 2023-09-30T01:07:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 10 client.4622.objecter ms_dispatch 0x7f1054001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 3 client.4622.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 3 client.4622.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f1059d16700 1 --2- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 0x7f103806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f1059d16700 10 monclient: get_auth_request con 0x7f1038068890 auth_method 0 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 20 client.4622.objecter dump_active .. 0 homeless 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f1059d16700 1 --2- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 0x7f103806ad40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f105413ebb0 tx=0x7f1044017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.450+0000 7f104a7fc700 10 client.4622.objecter ms_handle_connect 0x7f1038068890 2023-09-30T01:07:29.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.454+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1050051a90 con 0x7f10540a5c80 2023-09-30T01:07:29.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.454+0000 7f104a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:29.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.454+0000 7f104a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f105c77b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f105c77b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f105c77b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f105413d2e0 con 0x7f10540a5c80 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f104a7fc700 1 -- 172.21.15.115:0/1734383571 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f10500550b0 con 0x7f10540a5c80 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f104a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:29.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.676+0000 7f104a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:29.677 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:29.677 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 10 librados: watch_flush enter 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 10 librados: watch_flush exit 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 20 client.4622.objecter shutdown clearing up homeless session... 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 10 client.4622.objecter successfully canceled tick 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 msgr2=0x7f103806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 0x7f103806ad40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f105413ebb0 tx=0x7f1044017000 comp rx=0 tx=0).stop 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 10 monclient: shutdown 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 msgr2=0x7f10540a8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:29.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f105000ec80 tx=0x7f10500091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 shutdown_connections 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> v2:172.21.15.115:6816/115635 conn(0x7f1038068890 0x7f103806ad40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10540aecc0 0x7f105413d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f10540a5c80 0x7f10540a8470 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 --2- 172.21.15.115:0/1734383571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540ae8f0 0x7f105413d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 >> 172.21.15.115:0/1734383571 conn(0x7f105406a490 msgr2=0x7f10540630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 shutdown_connections 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 -- 172.21.15.115:0/1734383571 wait complete. 2023-09-30T01:07:29.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:29.678+0000 7f105c77b700 1 librados: shutdown 2023-09-30T01:07:32.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:32.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.817+0000 7ff8ab37e700 10 monclient: get_monmap_and_config 2023-09-30T01:07:32.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.817+0000 7ff8ab37e700 10 monclient: build_initial_monmap 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.817+0000 7ff8ab37e700 10 monclient: monmap: 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:32.819568+0000 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:32.819568+0000 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:32.819 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:32.820 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:32.820 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.818+0000 7ff8ab37e700 1 Processor -- start 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.818+0000 7ff8ab37e700 1 -- start start 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.818+0000 7ff8ab37e700 10 monclient: init 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.818+0000 7ff8ab37e700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.818+0000 7ff8ab37e700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8a40aaeb0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff8a40ab280 0x7ff8a4138390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-c con 0x7ff8a40ab280 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8a41388d0 0x7ff8a413b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-a con 0x7ff8a41388d0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a413aee0 con 0x7ff8a40aaeb0 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a406e930 con 0x7ff8a41388d0 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a406c110 con 0x7ff8a40ab280 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient(hunting): authenticate will time out at 1314.270073s 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8a41388d0 0x7ff8a413b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff8a41388d0 0x7ff8a413b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48160/0 (socket says 172.21.15.115:48160) 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 -- 172.21.15.115:0/3268361997 learned_addr learned my addr 172.21.15.115:0/3268361997 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:32.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): get_auth_request con 0x7ff8a41388d0 auth_method 0 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a8919700 1 --2- 172.21.15.115:0/3268361997 >> v2:172.21.15.157:3300/0 conn(0x7ff8a40ab280 0x7ff8a4138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a8919700 10 monclient(hunting): get_auth_request con 0x7ff8a40ab280 auth_method 0 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a8919700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a8919700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:32.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a8919700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient(hunting): handle_auth_done global_id 4854 payload 274 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 -- 172.21.15.115:0/3268361997 >> v2:172.21.15.157:3300/0 conn(0x7ff8a40ab280 msgr2=0x7ff8a4138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 --2- 172.21.15.115:0/3268361997 >> v2:172.21.15.157:3300/0 conn(0x7ff8a40ab280 0x7ff8a4138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 -- 172.21.15.115:0/3268361997 >> v1:172.21.15.157:6789/0 conn(0x7ff8a40aaeb0 legacy=0x7ff8a406f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient: _finish_hunting 0 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 monclient: found mon.noname-a 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 -- 172.21.15.115:0/3268361997 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a40acc90 con 0x7ff8a41388d0 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8a991b700 1 --2- 172.21.15.115:0/3268361997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a413b370 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4063c30 tx=0x7ff89c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23602f2472207a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 1 -- 172.21.15.115:0/3268361997 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c00e070 con 0x7ff8a41388d0 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:32.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: dump: 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: _finish_auth 0 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 1 -- 172.21.15.115:0/3268361997 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c00ba50 con 0x7ff8a41388d0 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 5 monclient: authenticate success, global_id 4854 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 1 -- 172.21.15.115:0/3268361997 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c0092e0 con 0x7ff8a41388d0 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff8ab37e700 4 set_mon_vals no callback set 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.819+0000 7ff89bfff700 10 monclient: dump: 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 10 monclient: shutdown 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3268361997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 msgr2=0x7ff8a413b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3268361997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a413b370 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4063c30 tx=0x7ff89c009840 comp rx=0 tx=0).stop 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3268361997 shutdown_connections 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3268361997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a413b370 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3268361997 >> v2:172.21.15.157:3300/0 conn(0x7ff8a40ab280 0x7ff8a4138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3268361997 >> 172.21.15.115:0/3268361997 conn(0x7ff8a40b0540 msgr2=0x7ff8a40b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3268361997 shutdown_connections 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.820+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3268361997 wait complete. 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 10 monclient: build_initial_monmap 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 10 monclient: monmap: 2023-09-30T01:07:32.835 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 librados: starting msgr at 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 librados: starting objecter 2023-09-30T01:07:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 Processor -- start 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 -- start start 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 librados: setting wanted keys 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 1 librados: calling monclient init 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.821+0000 7ff8ab37e700 10 monclient: init 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 0x7ff8a40a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-c con 0x7ff8a40aaeb0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 0x7ff8a413dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-b con 0x7ff8a40ab280 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): picked mon.noname-a con 0x7ff8a41388d0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40a7840 con 0x7ff8a41388d0 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40a7cd0 con 0x7ff8a40ab280 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a413ecb0 con 0x7ff8a40aaeb0 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8ab37e700 10 monclient(hunting): authenticate will time out at 1314.273258s 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 0x7ff8a40a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48174/0 (socket says 172.21.15.115:48174) 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 -- 172.21.15.115:0/3277857316 learned_addr learned my addr 172.21.15.115:0/3277857316 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): get_auth_request con 0x7ff8a41388d0 auth_method 0 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): get_auth_request con 0x7ff8a40aaeb0 auth_method 0 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:32.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a8919700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 0x7ff8a413dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient(hunting): handle_auth_done global_id 4857 payload 824 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 -- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 msgr2=0x7ff8a40a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 0x7ff8a40a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 -- 172.21.15.115:0/3277857316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 msgr2=0x7ff8a413dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 0x7ff8a413dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient: _finish_hunting 0 2023-09-30T01:07:32.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 monclient: found mon.noname-a 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 -- 172.21.15.115:0/3277857316 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c009ae0 con 0x7ff8a41388d0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a8919700 10 monclient: get_auth_request con 0x7ff8a40ab280 auth_method 0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a911a700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 0x7ff8a40a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a8919700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 0x7ff8a413dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8a991b700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8a406ae30 tx=0x7ff89c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 10 client.?.objecter ms_handle_connect 0x7ff8a41388d0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c007000 con 0x7ff8a41388d0 2023-09-30T01:07:32.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.822+0000 7ff8997fa700 10 monclient: dump: 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:32.842 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 10 monclient: _finish_auth 0 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c00be80 con 0x7ff8a41388d0 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 5 monclient: authenticate success, global_id 4857 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _renew_subs 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a40a7300 con 0x7ff8a41388d0 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c01c410 con 0x7ff8a41388d0 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:32.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8997fa700 10 monclient: dump: 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff89b7fe700 4 set_mon_vals no callback set 2023-09-30T01:07:32.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 client.4857.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _renew_subs 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a414ac30 con 0x7ff8a41388d0 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 1 librados: init done 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.823+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a40a79f0 con 0x7ff8a41388d0 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.824+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff89c003050 con 0x7ff8a41388d0 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 1 --2- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 0x7ff88806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:32.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff89c085750 con 0x7ff8a41388d0 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 10 client.4857.objecter ms_dispatch 0x7ff8a4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 3 client.4857.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 3 client.4857.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8a911a700 1 --2- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 0x7ff88806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8a911a700 10 monclient: get_auth_request con 0x7ff888068890 auth_method 0 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 20 client.4857.objecter dump_active .. 0 homeless 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8997fa700 10 client.4857.objecter ms_handle_connect 0x7ff888068890 2023-09-30T01:07:32.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.825+0000 7ff8a911a700 1 --2- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 0x7ff88806ad40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cfc0 tx=0x7ff8a0007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:32.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.829+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff89c053090 con 0x7ff8a41388d0 2023-09-30T01:07:32.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.829+0000 7ff8997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:32.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:32.829+0000 7ff8997fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.050+0000 7ff8ab37e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.050+0000 7ff8ab37e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.050+0000 7ff8ab37e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.050+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8a40a79f0 con 0x7ff8a41388d0 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.051+0000 7ff8997fa700 1 -- 172.21.15.115:0/3277857316 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff8a40a79f0 con 0x7ff8a41388d0 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.051+0000 7ff8997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:33.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.051+0000 7ff8997fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:33.053 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:33.053 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 10 librados: watch_flush enter 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 10 librados: watch_flush exit 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 20 client.4857.objecter shutdown clearing up homeless session... 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 10 client.4857.objecter successfully canceled tick 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 msgr2=0x7ff88806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 0x7ff88806ad40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cfc0 tx=0x7ff8a0007f50 comp rx=0 tx=0).stop 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 10 monclient: shutdown 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:33.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 msgr2=0x7ff8a414a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8a406ae30 tx=0x7ff89c018000 comp rx=0 tx=0).stop 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 shutdown_connections 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> v2:172.21.15.115:6816/115635 conn(0x7ff888068890 0x7ff88806ad40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8a41388d0 0x7ff8a414a530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff8a40aaeb0 0x7ff8a40a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 --2- 172.21.15.115:0/3277857316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8a40ab280 0x7ff8a413dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 >> 172.21.15.115:0/3277857316 conn(0x7ff8a40b0540 msgr2=0x7ff8a406c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 shutdown_connections 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 -- 172.21.15.115:0/3277857316 wait complete. 2023-09-30T01:07:33.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:33.053+0000 7ff8ab37e700 1 librados: shutdown 2023-09-30T01:07:36.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:36.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.194+0000 7fdb17e72700 10 monclient: get_monmap_and_config 2023-09-30T01:07:36.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.194+0000 7fdb17e72700 10 monclient: build_initial_monmap 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.194+0000 7fdb17e72700 10 monclient: monmap: 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:36.195321+0000 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:36.195321+0000 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:36.195 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:36.196 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:36.196 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.194+0000 7fdb17e72700 1 Processor -- start 2023-09-30T01:07:36.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 -- start start 2023-09-30T01:07:36.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient: init 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-b con 0x7fdb100ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fdb100aecc0 0x7fdb100b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-a con 0x7fdb100aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb101415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb100a7040 con 0x7fdb101415c0 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb100b19b0 con 0x7fdb100aecc0 2023-09-30T01:07:36.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb100a57f0 con 0x7fdb100ae8f0 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb17e72700 10 monclient(hunting): authenticate will time out at 1317.645883s 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.195+0000 7fdb15c0e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fdb100aecc0 0x7fdb100b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1640f700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fdb101415c0 0x7fdb100adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:36262/0 (socket says 172.21.15.115:36262) 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1640f700 1 -- 172.21.15.115:0/407156436 learned_addr learned my addr 172.21.15.115:0/407156436 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47242/0 (socket says 172.21.15.115:47242) 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): get_auth_request con 0x7fdb100aecc0 auth_method 0 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:36.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): get_auth_request con 0x7fdb100ae8f0 auth_method 0 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:36.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient(hunting): handle_auth_done global_id 4860 payload 274 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 -- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 msgr2=0x7fdb100a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 -- 172.21.15.115:0/407156436 >> v1:172.21.15.115:6789/0 conn(0x7fdb101415c0 legacy=0x7fdb100adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient: _finish_hunting 0 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 monclient: found mon.noname-a 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 -- 172.21.15.115:0/407156436 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb100a5480 con 0x7fdb100aecc0 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 1 -- 172.21.15.115:0/407156436 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815175219 0 0) 0x7fdb100a7040 con 0x7fdb101415c0 2023-09-30T01:07:36.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb1540d700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb100aecc0 0x7fdb100b1470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0098e0 tx=0x7fdb0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffed286e567c8bf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 1 -- 172.21.15.115:0/407156436 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00b640 con 0x7fdb100aecc0 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: dump: 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: _finish_auth 0 2023-09-30T01:07:36.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 1 -- 172.21.15.115:0/407156436 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c00b7a0 con 0x7fdb100aecc0 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb17e72700 5 monclient: authenticate success, global_id 4860 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb17e72700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb17e72700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 1 -- 172.21.15.115:0/407156436 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00bb40 con 0x7fdb100aecc0 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb17e72700 4 set_mon_vals no callback set 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.196+0000 7fdb14c0c700 10 monclient: dump: 2023-09-30T01:07:36.211 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 10 monclient: shutdown 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 -- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb100aecc0 msgr2=0x7fdb100b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb100aecc0 0x7fdb100b1470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0098e0 tx=0x7fdb0c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 -- 172.21.15.115:0/407156436 shutdown_connections 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100ae8f0 0x7fdb100a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 --2- 172.21.15.115:0/407156436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb100aecc0 0x7fdb100b1470 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 -- 172.21.15.115:0/407156436 >> 172.21.15.115:0/407156436 conn(0x7fdb1006a460 msgr2=0x7fdb100630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 -- 172.21.15.115:0/407156436 shutdown_connections 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 1 -- 172.21.15.115:0/407156436 wait complete. 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 10 monclient: build_initial_monmap 2023-09-30T01:07:36.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.197+0000 7fdb17e72700 10 monclient: monmap: 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:36.214 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 librados: starting msgr at 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 librados: starting objecter 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 Processor -- start 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 -- start start 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 librados: setting wanted keys 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 librados: calling monclient init 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient: init 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 0x7fdb1013eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-b con 0x7fdb100ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-c con 0x7fdb100aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): picked mon.noname-a con 0x7fdb101415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb100aab90 con 0x7fdb101415c0 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb10151340 con 0x7fdb100ae8f0 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb10151d10 con 0x7fdb100aecc0 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb17e72700 10 monclient(hunting): authenticate will time out at 1317.648875s 2023-09-30T01:07:36.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1540d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1540d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47246/0 (socket says 172.21.15.115:47246) 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48198/0 (socket says 172.21.15.115:48198) 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 1 -- 172.21.15.115:0/969606494 learned_addr learned my addr 172.21.15.115:0/969606494 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 10 monclient(hunting): get_auth_request con 0x7fdb101415c0 auth_method 0 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.198+0000 7fdb1640f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): get_auth_request con 0x7fdb100aecc0 auth_method 0 2023-09-30T01:07:36.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 0x7fdb1013eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient(hunting): handle_auth_done global_id 4866 payload 824 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 -- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 msgr2=0x7fdb101517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 -- 172.21.15.115:0/969606494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 msgr2=0x7fdb1013eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 0x7fdb1013eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient: _finish_hunting 0 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 monclient: found mon.noname-a 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 -- 172.21.15.115:0/969606494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00009420 con 0x7fdb101415c0 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb15c0e700 10 monclient: get_auth_request con 0x7fdb100ae8f0 auth_method 0 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 0x7fdb1013eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1540d700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb1640f700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800cbf0 tx=0x7fdb080019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 client.?.objecter ms_handle_connect 0x7fdb101415c0 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800a910 con 0x7fdb101415c0 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: dump: 2023-09-30T01:07:36.220 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: _finish_auth 0 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 5 monclient: authenticate success, global_id 4866 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0800aa70 con 0x7fdb101415c0 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800ae30 con 0x7fdb101415c0 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb05ffb700 10 monclient: dump: 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:36.222 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb07fff700 4 set_mon_vals no callback set 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 10 monclient: _renew_subs 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c009420 con 0x7fdb101415c0 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 10 client.4866.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 10 monclient: _renew_subs 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb100a9640 con 0x7fdb101415c0 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.199+0000 7fdb17e72700 1 librados: init done 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.200+0000 7fdb17e72700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.200+0000 7fdb17e72700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.200+0000 7fdb17e72700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.200+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb10072960 con 0x7fdb101415c0 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.200+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdb08007300 con 0x7fdb101415c0 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 1 --2- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 0x7fdaf406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdb08084220 con 0x7fdb101415c0 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 0x7fdaf406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 10 client.4866.objecter ms_dispatch 0x7fdb10001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 3 client.4866.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:36.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 3 client.4866.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:36.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb15c0e700 10 monclient: get_auth_request con 0x7fdaf4068890 auth_method 0 2023-09-30T01:07:36.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 20 client.4866.objecter dump_active .. 0 homeless 2023-09-30T01:07:36.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb15c0e700 1 --2- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 0x7fdaf406ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb000093f0 tx=0x7fdb00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:36.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.201+0000 7fdb05ffb700 10 client.4866.objecter ms_handle_connect 0x7fdaf4068890 2023-09-30T01:07:36.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.205+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb08051b60 con 0x7fdb101415c0 2023-09-30T01:07:36.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.205+0000 7fdb05ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:36.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.205+0000 7fdb05ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb17e72700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:36.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb17e72700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb17e72700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdb10151480 con 0x7fdb101415c0 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb05ffb700 1 -- 172.21.15.115:0/969606494 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdb08055180 con 0x7fdb101415c0 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb05ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.430+0000 7fdb05ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:36.431 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 10 librados: watch_flush enter 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 10 librados: watch_flush exit 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 20 client.4866.objecter shutdown clearing up homeless session... 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 10 client.4866.objecter successfully canceled tick 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 msgr2=0x7fdaf406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 0x7fdaf406ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb000093f0 tx=0x7fdb00008040 comp rx=0 tx=0).stop 2023-09-30T01:07:36.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 10 monclient: shutdown 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 msgr2=0x7fdb1014e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800cbf0 tx=0x7fdb080019b0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 shutdown_connections 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> v2:172.21.15.115:6816/115635 conn(0x7fdaf4068890 0x7fdaf406ad40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb101415c0 0x7fdb1014e1a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100ae8f0 0x7fdb1013eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 --2- 172.21.15.115:0/969606494 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fdb100aecc0 0x7fdb101517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.432+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 >> 172.21.15.115:0/969606494 conn(0x7fdb1006a460 msgr2=0x7fdb10140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.433+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 shutdown_connections 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.433+0000 7fdb17e72700 1 -- 172.21.15.115:0/969606494 wait complete. 2023-09-30T01:07:36.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:36.433+0000 7fdb17e72700 1 librados: shutdown 2023-09-30T01:07:39.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:39.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 10 monclient: get_monmap_and_config 2023-09-30T01:07:39.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 10 monclient: build_initial_monmap 2023-09-30T01:07:39.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 10 monclient: monmap: 2023-09-30T01:07:39.573 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:39.573 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:39.574071+0000 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:39.574071+0000 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:39.574 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:39.575 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:39.575 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 1 Processor -- start 2023-09-30T01:07:39.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 1 -- start start 2023-09-30T01:07:39.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.573+0000 7f1580183700 10 monclient: init 2023-09-30T01:07:39.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:39.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15780ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15780aecc0 0x7f15780a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): picked mon.noname-a con 0x7f15780aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): picked mon.noname-b con 0x7f15781415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780a7040 con 0x7f15780ae8f0 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15780a57f0 con 0x7f15780aecc0 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15780a5480 con 0x7f15781415c0 2023-09-30T01:07:39.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f1580183700 10 monclient(hunting): authenticate will time out at 1321.024615s 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15780aecc0 0x7f15780a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f15780aecc0 0x7f15780a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48204/0 (socket says 172.21.15.115:48204) 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47254/0 (socket says 172.21.15.115:47254) 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 1 -- 172.21.15.115:0/522107409 learned_addr learned my addr 172.21.15.115:0/522107409 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): get_auth_request con 0x7f15780aecc0 auth_method 0 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:39.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): get_auth_request con 0x7f15781415c0 auth_method 0 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.574+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 10 monclient(hunting): handle_auth_done global_id 4869 payload 274 2023-09-30T01:07:39.588 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 -- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 msgr2=0x7f15780adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 -- 172.21.15.115:0/522107409 >> v1:172.21.15.157:6789/0 conn(0x7f15780ae8f0 legacy=0x7f15780b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 10 monclient: _finish_hunting 0 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 monclient: found mon.noname-a 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 -- 172.21.15.115:0/522107409 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578141a40 con 0x7f15780aecc0 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157e720700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157d71e700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15780a5d10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f15680099b0 tx=0x7f1568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1266f986837448e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 1 -- 172.21.15.115:0/522107409 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568009ea0 con 0x7f15780aecc0 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:39.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: dump: 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: _finish_auth 0 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 5 monclient: authenticate success, global_id 4869 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 1 -- 172.21.15.115:0/522107409 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156800b860 con 0x7f15780aecc0 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 4 set_mon_vals no callback set 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 1 -- 172.21.15.115:0/522107409 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156800bc00 con 0x7f15780aecc0 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f157cf1d700 10 monclient: dump: 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 10 monclient: shutdown 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 -- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 msgr2=0x7f15780a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15780a5d10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f15680099b0 tx=0x7f1568009250 comp rx=0 tx=0).stop 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 -- 172.21.15.115:0/522107409 shutdown_connections 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f15780adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 --2- 172.21.15.115:0/522107409 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15780a5d10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.575+0000 7f1580183700 1 -- 172.21.15.115:0/522107409 >> 172.21.15.115:0/522107409 conn(0x7f157806a460 msgr2=0x7f15780630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 1 -- 172.21.15.115:0/522107409 shutdown_connections 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 1 -- 172.21.15.115:0/522107409 wait complete. 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 10 monclient: build_initial_monmap 2023-09-30T01:07:39.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 10 monclient: monmap: 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 1 librados: starting msgr at 2023-09-30T01:07:39.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 1 librados: starting objecter 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.576+0000 7f1580183700 1 Processor -- start 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 -- start start 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 librados: setting wanted keys 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 librados: calling monclient init 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient: init 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15780ae8f0 0x7f15780a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): picked mon.noname-b con 0x7f15780ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): picked mon.noname-a con 0x7f15780aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:39.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f157814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): picked mon.noname-c con 0x7f15781415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1578150510 con 0x7f15780aecc0 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15781501e0 con 0x7f15780ae8f0 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15780aac60 con 0x7f15781415c0 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f1580183700 10 monclient(hunting): authenticate will time out at 1321.027649s 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f157814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48212/0 (socket says 172.21.15.115:48212) 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 1 -- 172.21.15.115:0/2619570095 learned_addr learned my addr 172.21.15.115:0/2619570095 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): get_auth_request con 0x7f15781415c0 auth_method 0 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): get_auth_request con 0x7f15780aecc0 auth_method 0 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15780ae8f0 0x7f15780a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157e720700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): get_auth_request con 0x7f15780ae8f0 auth_method 0 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157d71e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:39.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.577+0000 7f157df1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 10 monclient(hunting): handle_auth_done global_id 4637 payload 824 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 -- 172.21.15.115:0/2619570095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15780ae8f0 msgr2=0x7f15780a9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15780ae8f0 0x7f15780a9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 -- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 msgr2=0x7f15781509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 10 monclient: _finish_hunting 0 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 monclient: found mon.noname-c 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:07:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 -- 172.21.15.115:0/2619570095 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570009a20 con 0x7f15781415c0 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157d71e700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f157e720700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f157814f160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1570009e70 tx=0x7f157000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 client.?.objecter ms_handle_connect 0x7f15781415c0 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157000caa0 con 0x7f15781415c0 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: dump: 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:39.600 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: _finish_auth 0 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157000f040 con 0x7f15781415c0 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 5 monclient: authenticate success, global_id 4637 2023-09-30T01:07:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 10 monclient: _renew_subs 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15680094f0 con 0x7f15781415c0 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157000ce90 con 0x7f15781415c0 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156dffb700 10 monclient: dump: 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:39.602 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f156ffff700 4 set_mon_vals no callback set 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 10 monclient: _renew_subs 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:39.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15780a7a40 con 0x7f15781415c0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.578+0000 7f1580183700 1 librados: init done 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.579+0000 7f1580183700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.579+0000 7f1580183700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.579+0000 7f1580183700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.579+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578150680 con 0x7f15781415c0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.579+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1570003020 con 0x7f15781415c0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 1 --2- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 0x7f155c06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f157df1f700 1 --2- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 0x7f155c06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1570084740 con 0x7f15781415c0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 10 client.4637.objecter ms_dispatch 0x7f1578001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 3 client.4637.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:39.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 3 client.4637.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f157df1f700 10 monclient: get_auth_request con 0x7f155c0688c0 auth_method 0 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 20 client.4637.objecter dump_active .. 0 homeless 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f157df1f700 1 --2- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 0x7f155c06ad70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15740039f0 tx=0x7f1574007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.580+0000 7f156dffb700 10 client.4637.objecter ms_handle_connect 0x7f155c0688c0 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.584+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1570051f60 con 0x7f15781415c0 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.584+0000 7f156dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:39.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.584+0000 7f156dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:39.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.806+0000 7f1580183700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.806+0000 7f1580183700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.806+0000 7f1580183700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.806+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1578072960 con 0x7f15781415c0 2023-09-30T01:07:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.807+0000 7f156dffb700 1 -- 172.21.15.115:0/2619570095 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1570055580 con 0x7f15781415c0 2023-09-30T01:07:39.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.807+0000 7f156dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:39.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.807+0000 7f156dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:39.810 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:39.811 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.808+0000 7f1580183700 10 librados: watch_flush enter 2023-09-30T01:07:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 10 librados: watch_flush exit 2023-09-30T01:07:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 20 client.4637.objecter shutdown clearing up homeless session... 2023-09-30T01:07:39.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 10 client.4637.objecter successfully canceled tick 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 msgr2=0x7f155c06ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 0x7f155c06ad70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15740039f0 tx=0x7f1574007ec0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 10 monclient: shutdown 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 msgr2=0x7f157814f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f157814f160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1570009e70 tx=0x7f157000b040 comp rx=0 tx=0).stop 2023-09-30T01:07:39.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 shutdown_connections 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> v2:172.21.15.115:6816/115635 conn(0x7f155c0688c0 0x7f155c06ad70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f15781415c0 0x7f157814f160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15780ae8f0 0x7f15780a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 --2- 172.21.15.115:0/2619570095 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15780aecc0 0x7f15781509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 >> 172.21.15.115:0/2619570095 conn(0x7f157806a460 msgr2=0x7f157813fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 shutdown_connections 2023-09-30T01:07:39.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 -- 172.21.15.115:0/2619570095 wait complete. 2023-09-30T01:07:39.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:39.809+0000 7f1580183700 1 librados: shutdown 2023-09-30T01:07:42.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.949+0000 7f3cc713c700 10 monclient: get_monmap_and_config 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.949+0000 7f3cc713c700 10 monclient: build_initial_monmap 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.949+0000 7f3cc713c700 10 monclient: monmap: 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:42.950909+0000 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:42.950909+0000 2023-09-30T01:07:42.950 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:42.951 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.950+0000 7f3cc713c700 1 Processor -- start 2023-09-30T01:07:42.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.950+0000 7f3cc713c700 1 -- start start 2023-09-30T01:07:42.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.950+0000 7f3cc713c700 10 monclient: init 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3cc00ae8f0 0x7f3cc00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-c con 0x7f3cc00ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cc00aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3cc01415c0 0x7f3cc00adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-a con 0x7f3cc01415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc00a7040 con 0x7f3cc00aecc0 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc00a57f0 con 0x7f3cc01415c0 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:42.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc00a5480 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc713c700 10 monclient(hunting): authenticate will time out at 1324.401428s 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3cc01415c0 0x7f3cc00adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f3cc01415c0 0x7f3cc00adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40430/0 (socket says 172.21.15.115:40430) 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 -- 172.21.15.115:0/2318118547 learned_addr learned my addr 172.21.15.115:0/2318118547 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): get_auth_request con 0x7f3cc01415c0 auth_method 0 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:42.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc4ed8700 1 --2- 172.21.15.115:0/2318118547 >> v2:172.21.15.157:3300/0 conn(0x7f3cc00ae8f0 0x7f3cc00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc4ed8700 10 monclient(hunting): get_auth_request con 0x7f3cc00ae8f0 auth_method 0 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc4ed8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc4ed8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc4ed8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient(hunting): handle_auth_done global_id 4875 payload 274 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 -- 172.21.15.115:0/2318118547 >> v2:172.21.15.157:3300/0 conn(0x7f3cc00ae8f0 msgr2=0x7f3cc00a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:42.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 --2- 172.21.15.115:0/2318118547 >> v2:172.21.15.157:3300/0 conn(0x7f3cc00ae8f0 0x7f3cc00a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 -- 172.21.15.115:0/2318118547 >> v1:172.21.15.157:6789/0 conn(0x7f3cc00aecc0 legacy=0x7f3cc00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient: _finish_hunting 0 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 monclient: found mon.noname-a 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.951+0000 7f3cc56d9700 1 -- 172.21.15.115:0/2318118547 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc00a55c0 con 0x7f3cc01415c0 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc56d9700 1 --2- 172.21.15.115:0/2318118547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc01415c0 0x7f3cc00adb60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00b59c0 tx=0x7f3cb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1c643b91801d45 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 1 -- 172.21.15.115:0/2318118547 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb800e070 con 0x7f3cc01415c0 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: dump: 2023-09-30T01:07:42.963 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:42.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 1 -- 172.21.15.115:0/2318118547 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb800ba50 con 0x7f3cc01415c0 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 5 monclient: authenticate success, global_id 4875 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 1 -- 172.21.15.115:0/2318118547 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb80092e0 con 0x7f3cc01415c0 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 4 set_mon_vals no callback set 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cbf7fe700 10 monclient: dump: 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.965 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 10 monclient: shutdown 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 -- 172.21.15.115:0/2318118547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc01415c0 msgr2=0x7f3cc00adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:42.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 --2- 172.21.15.115:0/2318118547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc01415c0 0x7f3cc00adb60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00b59c0 tx=0x7f3cb8009840 comp rx=0 tx=0).stop 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 -- 172.21.15.115:0/2318118547 shutdown_connections 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 --2- 172.21.15.115:0/2318118547 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc01415c0 0x7f3cc00adb60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 --2- 172.21.15.115:0/2318118547 >> v2:172.21.15.157:3300/0 conn(0x7f3cc00ae8f0 0x7f3cc00a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 -- 172.21.15.115:0/2318118547 >> 172.21.15.115:0/2318118547 conn(0x7f3cc006a460 msgr2=0x7f3cc00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 -- 172.21.15.115:0/2318118547 shutdown_connections 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.952+0000 7f3cc713c700 1 -- 172.21.15.115:0/2318118547 wait complete. 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 10 monclient: build_initial_monmap 2023-09-30T01:07:42.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 10 monclient: monmap: 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:42.968 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 librados: starting msgr at 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 librados: starting objecter 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 Processor -- start 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 -- start start 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 librados: setting wanted keys 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 1 librados: calling monclient init 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.953+0000 7f3cc713c700 10 monclient: init 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-a con 0x7f3cc00ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:42.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3cc00aecc0 0x7f3cc0151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-c con 0x7f3cc00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc01415c0 0x7f3cc00abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): picked mon.noname-b con 0x7f3cc01415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc00ab820 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0151e60 con 0x7f3cc01415c0 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0151b60 con 0x7f3cc00aecc0 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:42.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc713c700 10 monclient(hunting): authenticate will time out at 1324.404522s 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40438/0 (socket says 172.21.15.115:40438) 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 -- 172.21.15.115:0/77661722 learned_addr learned my addr 172.21.15.115:0/77661722 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): get_auth_request con 0x7f3cc00ae8f0 auth_method 0 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3cc00aecc0 0x7f3cc0151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): get_auth_request con 0x7f3cc00aecc0 auth_method 0 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:42.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cbffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc56d9700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc01415c0 0x7f3cc00abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc56d9700 10 monclient(hunting): get_auth_request con 0x7f3cc01415c0 auth_method 0 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc56d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc56d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc56d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient(hunting): handle_auth_done global_id 4878 payload 824 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 -- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3cc00aecc0 msgr2=0x7f3cc0151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3cc00aecc0 0x7f3cc0151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 -- 172.21.15.115:0/77661722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc01415c0 msgr2=0x7f3cc00abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc01415c0 0x7f3cc00abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient: _finish_hunting 0 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 monclient: found mon.noname-a 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.954+0000 7f3cc4ed8700 1 -- 172.21.15.115:0/77661722 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8009ae0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc4ed8700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d890 tx=0x7f3cb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:42.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 client.?.objecter ms_handle_connect 0x7f3cc00ae8f0 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb40125f0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: dump: 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:42.974 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: _finish_auth 0 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 5 monclient: authenticate success, global_id 4878 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4012750 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4012ad0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:42.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbcff9700 10 monclient: dump: 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cbeffd700 4 set_mon_vals no callback set 2023-09-30T01:07:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 10 monclient: _renew_subs 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc013eeb0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 10 client.4878.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 10 monclient: _renew_subs 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc00ac600 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.955+0000 7f3cc713c700 1 librados: init done 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.956+0000 7f3cc713c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.956+0000 7f3cc713c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.956+0000 7f3cc713c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.956+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc0152020 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.956+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3cb401e460 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 1 --2- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 0x7f3ca406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbffff700 1 --2- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 0x7f3ca406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbffff700 10 monclient: get_auth_request con 0x7f3ca4068890 auth_method 0 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3cb40844f0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 10 client.4878.objecter ms_dispatch 0x7f3cc0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 3 client.4878.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 3 client.4878.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 20 client.4878.objecter dump_active .. 0 homeless 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbcff9700 10 client.4878.objecter ms_handle_connect 0x7f3ca4068890 2023-09-30T01:07:42.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.957+0000 7f3cbffff700 1 --2- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 0x7f3ca406ad40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3cc013ff40 tx=0x7f3cac007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:42.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.961+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cb4051db0 con 0x7f3cc00ae8f0 2023-09-30T01:07:42.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.961+0000 7f3cbcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:42.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:42.961+0000 7f3cbcff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.183+0000 7f3cc713c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.183+0000 7f3cc713c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.183+0000 7f3cc713c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.183+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3cc00abcb0 con 0x7f3cc00ae8f0 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.184+0000 7f3cbcff9700 1 -- 172.21.15.115:0/77661722 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3cb40553d0 con 0x7f3cc00ae8f0 2023-09-30T01:07:43.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.184+0000 7f3cbcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:43.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.184+0000 7f3cbcff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:43.185 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:43.185 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:43.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 10 librados: watch_flush enter 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 10 librados: watch_flush exit 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 20 client.4878.objecter shutdown clearing up homeless session... 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 10 client.4878.objecter successfully canceled tick 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 msgr2=0x7f3ca406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 0x7f3ca406ad40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3cc013ff40 tx=0x7f3cac007ec0 comp rx=0 tx=0).stop 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 10 monclient: shutdown 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 msgr2=0x7f3cc013e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d890 tx=0x7f3cb4008040 comp rx=0 tx=0).stop 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 shutdown_connections 2023-09-30T01:07:43.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> v2:172.21.15.115:6816/115635 conn(0x7f3ca4068890 0x7f3ca406ad40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc01415c0 0x7f3cc00abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cc00ae8f0 0x7f3cc013e230 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 --2- 172.21.15.115:0/77661722 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3cc00aecc0 0x7f3cc0151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 >> 172.21.15.115:0/77661722 conn(0x7f3cc006a460 msgr2=0x7f3cc00b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 shutdown_connections 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 -- 172.21.15.115:0/77661722 wait complete. 2023-09-30T01:07:43.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:43.186+0000 7f3cc713c700 1 librados: shutdown 2023-09-30T01:07:46.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.326+0000 7f3de64b8700 10 monclient: get_monmap_and_config 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.326+0000 7f3de64b8700 10 monclient: build_initial_monmap 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.326+0000 7f3de64b8700 10 monclient: monmap: 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:46.327634+0000 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:46.327634+0000 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:46.328 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.327+0000 7f3de64b8700 1 Processor -- start 2023-09-30T01:07:46.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.327+0000 7f3de64b8700 1 -- start start 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.327+0000 7f3de64b8700 10 monclient: init 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.327+0000 7f3de64b8700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.327+0000 7f3de64b8700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 0x7f3de00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-b con 0x7f3de00ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3de00aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 0x7f3de00adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-c con 0x7f3de01415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a7040 con 0x7f3de00aecc0 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de00a57f0 con 0x7f3de01415c0 2023-09-30T01:07:46.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a5480 con 0x7f3de00ae8f0 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient(hunting): authenticate will time out at 1327.778077s 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 0x7f3de00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddf7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f3de00aecc0 0x7f3de00b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56352/0 (socket says 172.21.15.115:56352) 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/4045142754 learned_addr learned my addr 172.21.15.115:0/4045142754 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): get_auth_request con 0x7f3de00ae8f0 auth_method 0 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:46.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 1 --2- 172.21.15.115:0/4045142754 >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 0x7f3de00adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 10 monclient(hunting): get_auth_request con 0x7f3de01415c0 auth_method 0 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 1 -- 172.21.15.115:0/4045142754 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193072052 0 0) 0x7f3de00a7040 con 0x7f3de00aecc0 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient(hunting): handle_auth_done global_id 4643 payload 274 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 -- 172.21.15.115:0/4045142754 >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 msgr2=0x7f3de00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 --2- 172.21.15.115:0/4045142754 >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 0x7f3de00adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 -- 172.21.15.115:0/4045142754 >> v1:172.21.15.115:6789/0 conn(0x7f3de00aecc0 legacy=0x7f3de00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient: _finish_hunting 0 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 monclient: found mon.noname-b 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 -- 172.21.15.115:0/4045142754 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00a55c0 con 0x7f3de00ae8f0 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de4a55700 1 --2- 172.21.15.115:0/4045142754 >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 0x7f3de00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddffff700 1 --2- 172.21.15.115:0/4045142754 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 0x7f3de00a5d10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dc80098e0 tx=0x7f3dc80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=761adf31e3fea77 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 1 -- 172.21.15.115:0/4045142754 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8009e80 con 0x7f3de00ae8f0 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: dump: 2023-09-30T01:07:46.340 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: _finish_auth 0 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:46.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 5 monclient: authenticate success, global_id 4643 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 1 -- 172.21.15.115:0/4045142754 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc800b860 con 0x7f3de00ae8f0 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3de64b8700 4 set_mon_vals no callback set 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 1 -- 172.21.15.115:0/4045142754 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc800bbe0 con 0x7f3de00ae8f0 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.328+0000 7f3ddeffd700 10 monclient: dump: 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:46.342 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 10 monclient: shutdown 2023-09-30T01:07:46.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 -- 172.21.15.115:0/4045142754 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 msgr2=0x7f3de00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 --2- 172.21.15.115:0/4045142754 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 0x7f3de00a5d10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dc80098e0 tx=0x7f3dc80091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 -- 172.21.15.115:0/4045142754 shutdown_connections 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 --2- 172.21.15.115:0/4045142754 >> v2:172.21.15.157:3300/0 conn(0x7f3de01415c0 0x7f3de00adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 --2- 172.21.15.115:0/4045142754 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de00ae8f0 0x7f3de00a5d10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 -- 172.21.15.115:0/4045142754 >> 172.21.15.115:0/4045142754 conn(0x7f3de006a460 msgr2=0x7f3de00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 -- 172.21.15.115:0/4045142754 shutdown_connections 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 1 -- 172.21.15.115:0/4045142754 wait complete. 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 10 monclient: build_initial_monmap 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.329+0000 7f3de64b8700 10 monclient: monmap: 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:46.344 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 librados: starting msgr at 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 librados: starting objecter 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 Processor -- start 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 -- start start 2023-09-30T01:07:46.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 librados: setting wanted keys 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 1 librados: calling monclient init 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.330+0000 7f3de64b8700 10 monclient: init 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 0x7f3de013ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-b con 0x7f3de00ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-a con 0x7f3de00aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:46.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): picked mon.noname-c con 0x7f3de01415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00ad8d0 con 0x7f3de00aecc0 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00aa1f0 con 0x7f3de00ae8f0 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a88a0 con 0x7f3de01415c0 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient(hunting): authenticate will time out at 1327.781158s 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58406/0 (socket says 172.21.15.115:58406) 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40442/0 (socket says 172.21.15.115:40442) 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 -- 172.21.15.115:0/2692097558 learned_addr learned my addr 172.21.15.115:0/2692097558 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): get_auth_request con 0x7f3de01415c0 auth_method 0 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): get_auth_request con 0x7f3de00aecc0 auth_method 0 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:46.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddffff700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 0x7f3de013ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient(hunting): handle_auth_done global_id 4646 payload 824 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 -- 172.21.15.115:0/2692097558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 msgr2=0x7f3de013ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 0x7f3de013ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 -- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 msgr2=0x7f3de014ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient: _finish_hunting 0 2023-09-30T01:07:46.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 monclient: found mon.noname-c 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 -- 172.21.15.115:0/2692097558 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8009420 con 0x7f3de01415c0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddffff700 10 monclient: get_auth_request con 0x7f3de00ae8f0 auth_method 0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddffff700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 0x7f3de013ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3ddf7fe700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de4a55700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009fc0 tx=0x7f3dd80099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 client.?.objecter ms_handle_connect 0x7f3de01415c0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800dc90 con 0x7f3de01415c0 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:46.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: dump: 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:46.351 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: _finish_auth 0 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 5 monclient: authenticate success, global_id 4646 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd800f040 con 0x7f3de01415c0 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _renew_subs 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de00a9e10 con 0x7f3de01415c0 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8015690 con 0x7f3de01415c0 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:46.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dd7fff700 10 monclient: dump: 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:46.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3dde7fc700 4 set_mon_vals no callback set 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _renew_subs 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de00a96a0 con 0x7f3de01415c0 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.331+0000 7f3de64b8700 1 librados: init done 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.332+0000 7f3de64b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.332+0000 7f3de64b8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.332+0000 7f3de64b8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.332+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de00a6fb0 con 0x7f3de01415c0 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3dd801e3b0 con 0x7f3de01415c0 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 1 --2- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 0x7f3dc006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:46.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3dd8084390 con 0x7f3de01415c0 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 10 client.4646.objecter ms_dispatch 0x7f3de0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 3 client.4646.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 3 client.4646.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3ddffff700 1 --2- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 0x7f3dc006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3ddffff700 10 monclient: get_auth_request con 0x7f3dc0068890 auth_method 0 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.333+0000 7f3dd7fff700 20 client.4646.objecter dump_active .. 0 homeless 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.334+0000 7f3dd7fff700 10 client.4646.objecter ms_handle_connect 0x7f3dc0068890 2023-09-30T01:07:46.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.334+0000 7f3ddffff700 1 --2- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 0x7f3dc006ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8003080 tx=0x7f3dc8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:46.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.338+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3dd8051c50 con 0x7f3de01415c0 2023-09-30T01:07:46.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.338+0000 7f3dd7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:46.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.338+0000 7f3dd7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3de64b8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3de64b8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3de64b8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3de0072960 con 0x7f3de01415c0 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3dd7fff700 1 -- 172.21.15.115:0/2692097558 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3dd801a100 con 0x7f3de01415c0 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3dd7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.563+0000 7f3dd7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:46.564 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:46.565 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 10 librados: watch_flush enter 2023-09-30T01:07:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 10 librados: watch_flush exit 2023-09-30T01:07:46.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 20 client.4646.objecter shutdown clearing up homeless session... 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 10 client.4646.objecter successfully canceled tick 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 msgr2=0x7f3dc006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 0x7f3dc006ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8003080 tx=0x7f3dc8017000 comp rx=0 tx=0).stop 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 10 monclient: shutdown 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 msgr2=0x7f3de00a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009fc0 tx=0x7f3dd80099b0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 shutdown_connections 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> v2:172.21.15.115:6816/115635 conn(0x7f3dc0068890 0x7f3dc006ad40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3de01415c0 0x7f3de00a7760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de00ae8f0 0x7f3de013ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 --2- 172.21.15.115:0/2692097558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3de00aecc0 0x7f3de014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.565+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 >> 172.21.15.115:0/2692097558 conn(0x7f3de006a460 msgr2=0x7f3de00b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.566+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 shutdown_connections 2023-09-30T01:07:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.566+0000 7f3de64b8700 1 -- 172.21.15.115:0/2692097558 wait complete. 2023-09-30T01:07:46.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:46.566+0000 7f3de64b8700 1 librados: shutdown 2023-09-30T01:07:49.579 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 10 monclient: get_monmap_and_config 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 10 monclient: build_initial_monmap 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 10 monclient: monmap: 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:49.705 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:49.706170+0000 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:49.706170+0000 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:49.706 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 1 Processor -- start 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 1 -- start start 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.705+0000 7f2fdb2d9700 10 monclient: init 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-b con 0x7f2fd40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fd40aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2fd41415c0 0x7f2fd40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-a con 0x7f2fd41415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40a7040 con 0x7f2fd40aecc0 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40a57f0 con 0x7f2fd41415c0 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40a5480 con 0x7f2fd40ae8f0 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fdb2d9700 10 monclient(hunting): authenticate will time out at 1331.156580s 2023-09-30T01:07:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2fd41415c0 0x7f2fd40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd8874700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f2fd40aecc0 0x7f2fd40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:56368/0 (socket says 172.21.15.115:56368) 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2fd41415c0 0x7f2fd40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40444/0 (socket says 172.21.15.115:40444) 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:58420/0 (socket says 172.21.15.115:58420) 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 1 -- 172.21.15.115:0/1401838421 learned_addr learned my addr 172.21.15.115:0/1401838421 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): get_auth_request con 0x7f2fd41415c0 auth_method 0 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): get_auth_request con 0x7f2fd40ae8f0 auth_method 0 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.706+0000 7f2fd9876700 10 monclient(hunting): handle_auth_done global_id 4887 payload 274 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 -- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 msgr2=0x7f2fd40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 -- 172.21.15.115:0/1401838421 >> v1:172.21.15.115:6789/0 conn(0x7f2fd40aecc0 legacy=0x7f2fd40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 10 monclient: _finish_hunting 0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 monclient: found mon.noname-a 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 1 -- 172.21.15.115:0/1401838421 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017582673 0 0) 0x7f2fd40a7040 con 0x7f2fd40aecc0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 -- 172.21.15.115:0/1401838421 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40a55c0 con 0x7f2fd41415c0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9075700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fd9876700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40adb60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009f20 tx=0x7f2fcc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72e1f62cd6828d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 1 -- 172.21.15.115:0/1401838421 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00e070 con 0x7f2fd41415c0 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:49.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: dump: 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:49.720 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: _finish_auth 0 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 1 -- 172.21.15.115:0/1401838421 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc00b9d0 con 0x7f2fd41415c0 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 5 monclient: authenticate success, global_id 4887 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 1 -- 172.21.15.115:0/1401838421 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0092e0 con 0x7f2fd41415c0 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 4 set_mon_vals no callback set 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fcbfff700 10 monclient: dump: 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 10 monclient: shutdown 2023-09-30T01:07:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 msgr2=0x7f2fd40adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40adb60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009f20 tx=0x7f2fcc009840 comp rx=0 tx=0).stop 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/1401838421 shutdown_connections 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40ae8f0 0x7f2fd40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/1401838421 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40adb60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.707+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/1401838421 >> 172.21.15.115:0/1401838421 conn(0x7f2fd406a460 msgr2=0x7f2fd40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/1401838421 shutdown_connections 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/1401838421 wait complete. 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 10 monclient: build_initial_monmap 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 10 monclient: monmap: 2023-09-30T01:07:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 1 librados: starting msgr at 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 1 librados: starting objecter 2023-09-30T01:07:49.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.708+0000 7f2fdb2d9700 1 Processor -- start 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 -- start start 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 librados: setting wanted keys 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 librados: calling monclient init 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient: init 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fd40ae8f0 0x7f2fd413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-b con 0x7f2fd40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 0x7f2fd40a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-c con 0x7f2fd40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): picked mon.noname-a con 0x7f2fd41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd413e900 con 0x7f2fd41415c0 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40a83c0 con 0x7f2fd40ae8f0 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40a86d0 con 0x7f2fd40aecc0 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fdb2d9700 10 monclient(hunting): authenticate will time out at 1331.159701s 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 0x7f2fd40a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40448/0 (socket says 172.21.15.115:40448) 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 1 -- 172.21.15.115:0/4170869479 learned_addr learned my addr 172.21.15.115:0/4170869479 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): get_auth_request con 0x7f2fd40aecc0 auth_method 0 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): get_auth_request con 0x7f2fd41415c0 auth_method 0 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd8874700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9075700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fd40ae8f0 0x7f2fd413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9876700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9075700 10 monclient(hunting): get_auth_request con 0x7f2fd40ae8f0 auth_method 0 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9075700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9075700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.709+0000 7f2fd9075700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9075700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 -- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 msgr2=0x7f2fd40a7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 0x7f2fd40a7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 -- 172.21.15.115:0/4170869479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fd40ae8f0 msgr2=0x7f2fd413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fd40ae8f0 0x7f2fd413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 10 monclient: _finish_hunting 0 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 monclient: found mon.noname-a 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 -- 172.21.15.115:0/4170869479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4009420 con 0x7f2fd41415c0 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd8874700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 0x7f2fd40a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fd9876700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40b1dd0 tx=0x7f2fcc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 client.?.objecter ms_handle_connect 0x7f2fd41415c0 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc008b10 con 0x7f2fd41415c0 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: dump: 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: _finish_auth 0 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc008c70 con 0x7f2fd41415c0 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 5 monclient: authenticate success, global_id 4893 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 10 monclient: _renew_subs 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc009ae0 con 0x7f2fd41415c0 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc025460 con 0x7f2fd41415c0 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fc97fa700 10 monclient: dump: 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fcb7fe700 4 set_mon_vals no callback set 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 10 monclient: _renew_subs 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd40a9850 con 0x7f2fd41415c0 2023-09-30T01:07:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.710+0000 7f2fdb2d9700 1 librados: init done 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.711+0000 7f2fdb2d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.711+0000 7f2fdb2d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.711+0000 7f2fdb2d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.711+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4072960 con 0x7f2fd41415c0 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.711+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2fcc003040 con 0x7f2fd41415c0 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 1 --2- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 0x7f2fb806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2fcc0859b0 con 0x7f2fd41415c0 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 10 client.4893.objecter ms_dispatch 0x7f2fd4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 3 client.4893.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 3 client.4893.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fd9075700 1 --2- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 0x7f2fb806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fd9075700 10 monclient: get_auth_request con 0x7f2fb8068890 auth_method 0 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 20 client.4893.objecter dump_active .. 0 homeless 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fc97fa700 10 client.4893.objecter ms_handle_connect 0x7f2fb8068890 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.712+0000 7f2fd9075700 1 --2- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 0x7f2fb806ad40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009f40 tx=0x7f2fc40091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.716+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fcc053270 con 0x7f2fd41415c0 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.716+0000 7f2fc97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.716+0000 7f2fc97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.938+0000 7f2fdb2d9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.938+0000 7f2fdb2d9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.938+0000 7f2fdb2d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.938+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fd4141a40 con 0x7f2fd41415c0 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.939+0000 7f2fc97fa700 1 -- 172.21.15.115:0/4170869479 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2fcc056890 con 0x7f2fd41415c0 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.939+0000 7f2fc97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.939+0000 7f2fc97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:49.940 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:49.940 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 10 librados: watch_flush enter 2023-09-30T01:07:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 10 librados: watch_flush exit 2023-09-30T01:07:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 20 client.4893.objecter shutdown clearing up homeless session... 2023-09-30T01:07:49.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 10 client.4893.objecter successfully canceled tick 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 msgr2=0x7f2fb806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 0x7f2fb806ad40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009f40 tx=0x7f2fc40091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 10 monclient: shutdown 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 msgr2=0x7f2fd40a9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40b1dd0 tx=0x7f2fcc018000 comp rx=0 tx=0).stop 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 shutdown_connections 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> v2:172.21.15.115:6816/115635 conn(0x7f2fb8068890 0x7f2fb806ad40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fd41415c0 0x7f2fd40a9310 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fd40ae8f0 0x7f2fd413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 --2- 172.21.15.115:0/4170869479 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2fd40aecc0 0x7f2fd40a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 >> 172.21.15.115:0/4170869479 conn(0x7f2fd406a460 msgr2=0x7f2fd4140920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 shutdown_connections 2023-09-30T01:07:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 -- 172.21.15.115:0/4170869479 wait complete. 2023-09-30T01:07:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:49.941+0000 7f2fdb2d9700 1 librados: shutdown 2023-09-30T01:07:52.953 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:53.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 10 monclient: get_monmap_and_config 2023-09-30T01:07:53.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 10 monclient: build_initial_monmap 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 10 monclient: monmap: 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:53.083089+0000 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:53.083089+0000 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:53.083 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:53.084 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:53.084 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 1 Processor -- start 2023-09-30T01:07:53.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 1 -- start start 2023-09-30T01:07:53.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.082+0000 7fefb8ec7700 10 monclient: init 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fefb40ae8f0 0x7fefb40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-c con 0x7fefb40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefb40aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-b con 0x7fefb41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40a7040 con 0x7fefb40aecc0 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb40a57f0 con 0x7fefb40ae8f0 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40a5480 con 0x7fefb41415c0 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb8ec7700 10 monclient(hunting): authenticate will time out at 1334.533543s 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:54358/0 (socket says 172.21.15.115:54358) 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 -- 172.21.15.115:0/2414349086 learned_addr learned my addr 172.21.15.115:0/2414349086 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:53.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): get_auth_request con 0x7fefb41415c0 auth_method 0 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb259c700 1 --2- 172.21.15.115:0/2414349086 >> v2:172.21.15.157:3300/0 conn(0x7fefb40ae8f0 0x7fefb40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb259c700 10 monclient(hunting): get_auth_request con 0x7fefb40ae8f0 auth_method 0 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_done global_id 4655 payload 274 2023-09-30T01:07:53.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 -- 172.21.15.115:0/2414349086 >> v2:172.21.15.157:3300/0 conn(0x7fefb40ae8f0 msgr2=0x7fefb40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 --2- 172.21.15.115:0/2414349086 >> v2:172.21.15.157:3300/0 conn(0x7fefb40ae8f0 0x7fefb40a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 -- 172.21.15.115:0/2414349086 >> v1:172.21.15.157:6789/0 conn(0x7fefb40aecc0 legacy=0x7fefb40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 monclient: found mon.noname-b 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.083+0000 7fefb2d9d700 1 -- 172.21.15.115:0/2414349086 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4141a40 con 0x7fefb41415c0 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb2d9d700 1 --2- 172.21.15.115:0/2414349086 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefac009f20 tx=0x7fefac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb76ca21d9c9977e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 1 -- 172.21.15.115:0/2414349086 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac00e070 con 0x7fefb41415c0 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:53.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: dump: 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: _finish_auth 0 2023-09-30T01:07:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 5 monclient: authenticate success, global_id 4655 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 1 -- 172.21.15.115:0/2414349086 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac00ba80 con 0x7fefb41415c0 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 1 -- 172.21.15.115:0/2414349086 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac0092e0 con 0x7fefb41415c0 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 4 set_mon_vals no callback set 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb159a700 10 monclient: dump: 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:53.099 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 10 monclient: shutdown 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 -- 172.21.15.115:0/2414349086 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 msgr2=0x7fefb40adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.100 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/2414349086 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefac009f20 tx=0x7fefac009840 comp rx=0 tx=0).stop 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 -- 172.21.15.115:0/2414349086 shutdown_connections 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/2414349086 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb40adb60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/2414349086 >> v2:172.21.15.157:3300/0 conn(0x7fefb40ae8f0 0x7fefb40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 -- 172.21.15.115:0/2414349086 >> 172.21.15.115:0/2414349086 conn(0x7fefb406a460 msgr2=0x7fefb40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 -- 172.21.15.115:0/2414349086 shutdown_connections 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.084+0000 7fefb8ec7700 1 -- 172.21.15.115:0/2414349086 wait complete. 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.085+0000 7fefb8ec7700 10 monclient: build_initial_monmap 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.085+0000 7fefb8ec7700 10 monclient: monmap: 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 librados: starting msgr at 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 librados: starting objecter 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 Processor -- start 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 -- start start 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 librados: setting wanted keys 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 librados: calling monclient init 2023-09-30T01:07:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 10 monclient: init 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefb40ae8f0 0x7fefb413e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-b con 0x7fefb40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.086+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-a con 0x7fefb40aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb4151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): picked mon.noname-c con 0x7fefb41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb4150fa0 con 0x7fefb40aecc0 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40aba20 con 0x7fefb40ae8f0 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40a7760 con 0x7fefb41415c0 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient(hunting): authenticate will time out at 1334.537050s 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52138/0 (socket says 172.21.15.115:52138) 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 -- 172.21.15.115:0/3713164010 learned_addr learned my addr 172.21.15.115:0/3713164010 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): get_auth_request con 0x7fefb40aecc0 auth_method 0 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:53.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefb40ae8f0 0x7fefb413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): get_auth_request con 0x7fefb40ae8f0 auth_method 0 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb4151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): get_auth_request con 0x7fefb41415c0 auth_method 0 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient(hunting): handle_auth_done global_id 4896 payload 824 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 -- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 msgr2=0x7fefb4151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb4151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 -- 172.21.15.115:0/3713164010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefb40ae8f0 msgr2=0x7fefb413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefb40ae8f0 0x7fefb413e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 monclient: found mon.noname-a 2023-09-30T01:07:53.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 -- 172.21.15.115:0/3713164010 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009ae0 con 0x7fefb40aecc0 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb2d9d700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb1d9b700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c5c0 tx=0x7fefa400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 client.?.objecter ms_handle_connect 0x7fefb40aecc0 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4011590 con 0x7fefb40aecc0 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: dump: 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:53.107 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:53.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa4011710 con 0x7fefb40aecc0 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 5 monclient: authenticate success, global_id 4896 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient: _renew_subs 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb41508b0 con 0x7fefb40aecc0 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4011ad0 con 0x7fefb40aecc0 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefaa7fc700 10 monclient: dump: 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:53.109 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb0d99700 4 set_mon_vals no callback set 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 client.4896.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient: _renew_subs 2023-09-30T01:07:53.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb4151970 con 0x7fefb40aecc0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.087+0000 7fefb8ec7700 1 librados: init done 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.088+0000 7fefb8ec7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.088+0000 7fefb8ec7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.088+0000 7fefb8ec7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.088+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb4151150 con 0x7fefb40aecc0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fefa401b3b0 con 0x7fefb40aecc0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 1 --2- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 0x7fef9406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fefa4083360 con 0x7fefb40aecc0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 10 client.4896.objecter ms_dispatch 0x7fefb4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 3 client.4896.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 3 client.4896.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:53.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefb259c700 1 --2- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 0x7fef9406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefb259c700 10 monclient: get_auth_request con 0x7fef94068890 auth_method 0 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 20 client.4896.objecter dump_active .. 0 homeless 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefaa7fc700 10 client.4896.objecter ms_handle_connect 0x7fef94068890 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.089+0000 7fefb259c700 1 --2- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 0x7fef9406ad40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef9c006fd0 tx=0x7fef9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.093+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefb4151150 con 0x7fefb40aecc0 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.093+0000 7fefaa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:53.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.093+0000 7fefaa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.313+0000 7fefb8ec7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.313+0000 7fefb8ec7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.313+0000 7fefb8ec7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.313+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fefb4151150 con 0x7fefb40aecc0 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.314+0000 7fefaa7fc700 1 -- 172.21.15.115:0/3713164010 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fefb4151150 con 0x7fefb40aecc0 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.314+0000 7fefaa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.314+0000 7fefaa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:53.314 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:53.315 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.315+0000 7fefb8ec7700 10 librados: watch_flush enter 2023-09-30T01:07:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.315+0000 7fefb8ec7700 10 librados: watch_flush exit 2023-09-30T01:07:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.315+0000 7fefb8ec7700 20 client.4896.objecter shutdown clearing up homeless session... 2023-09-30T01:07:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.315+0000 7fefb8ec7700 10 client.4896.objecter successfully canceled tick 2023-09-30T01:07:53.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 msgr2=0x7fef9406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 0x7fef9406ad40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef9c006fd0 tx=0x7fef9c008040 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 10 monclient: shutdown 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 msgr2=0x7fefb4150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c5c0 tx=0x7fefa400c8d0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 shutdown_connections 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> v2:172.21.15.115:6816/115635 conn(0x7fef94068890 0x7fef9406ad40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fefb41415c0 0x7fefb4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefb40ae8f0 0x7fefb413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 --2- 172.21.15.115:0/3713164010 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fefb40aecc0 0x7fefb4150370 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 >> 172.21.15.115:0/3713164010 conn(0x7fefb406a460 msgr2=0x7fefb413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:53.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 shutdown_connections 2023-09-30T01:07:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 -- 172.21.15.115:0/3713164010 wait complete. 2023-09-30T01:07:53.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:53.316+0000 7fefb8ec7700 1 librados: shutdown 2023-09-30T01:07:56.330 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:56.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.458+0000 7fb43a953700 10 monclient: get_monmap_and_config 2023-09-30T01:07:56.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.458+0000 7fb43a953700 10 monclient: build_initial_monmap 2023-09-30T01:07:56.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.458+0000 7fb43a953700 10 monclient: monmap: 2023-09-30T01:07:56.458 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:56.459322+0000 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:56.459322+0000 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:56.459 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 Processor -- start 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 -- start start 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient: init 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb4340ae8f0 0x7fb4340a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-c con 0x7fb4340ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4340aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb4341415c0 0x7fb4340adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-a con 0x7fb4341415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4340a7040 con 0x7fb4340aecc0 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340a57f0 con 0x7fb4341415c0 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340a5480 con 0x7fb4340ae8f0 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:56.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb43a953700 10 monclient(hunting): authenticate will time out at 1337.909884s 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.459+0000 7fb4337fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fb4340aecc0 0x7fb4340b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57844/0 (socket says 172.21.15.115:57844) 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb4337fe700 1 -- 172.21.15.115:0/1768371160 learned_addr learned my addr 172.21.15.115:0/1768371160 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 --2- 172.21.15.115:0/1768371160 >> v2:172.21.15.115:3300/0 conn(0x7fb4341415c0 0x7fb4340adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): get_auth_request con 0x7fb4341415c0 auth_method 0 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 1 --2- 172.21.15.115:0/1768371160 >> v2:172.21.15.157:3300/0 conn(0x7fb4340ae8f0 0x7fb4340a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): get_auth_request con 0x7fb4340ae8f0 auth_method 0 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:56.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient(hunting): handle_auth_done global_id 4899 payload 274 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 -- 172.21.15.115:0/1768371160 >> v2:172.21.15.157:3300/0 conn(0x7fb4340ae8f0 msgr2=0x7fb4340a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 --2- 172.21.15.115:0/1768371160 >> v2:172.21.15.157:3300/0 conn(0x7fb4340ae8f0 0x7fb4340a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 -- 172.21.15.115:0/1768371160 >> v1:172.21.15.115:6789/0 conn(0x7fb4340aecc0 legacy=0x7fb4340b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient: _finish_hunting 0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 monclient: found mon.noname-a 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 -- 172.21.15.115:0/1768371160 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340a55c0 con 0x7fb4341415c0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 1 -- 172.21.15.115:0/1768371160 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472858205 0 0) 0x7fb4340a7040 con 0x7fb4340aecc0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb438ef0700 1 --2- 172.21.15.115:0/1768371160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4341415c0 0x7fb4340adb60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb4340b59c0 tx=0x7fb42c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c4e2f72d10627d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 1 -- 172.21.15.115:0/1768371160 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e070 con 0x7fb4341415c0 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:56.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: dump: 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: _finish_auth 0 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:56.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb43a953700 5 monclient: authenticate success, global_id 4899 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb43a953700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 1 -- 172.21.15.115:0/1768371160 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00bb60 con 0x7fb4341415c0 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb43a953700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb43a953700 4 set_mon_vals no callback set 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 1 -- 172.21.15.115:0/1768371160 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c012430 con 0x7fb4341415c0 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.460+0000 7fb432ffd700 10 monclient: dump: 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 10 monclient: shutdown 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 -- 172.21.15.115:0/1768371160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4341415c0 msgr2=0x7fb4340adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 --2- 172.21.15.115:0/1768371160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4341415c0 0x7fb4340adb60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb4340b59c0 tx=0x7fb42c009780 comp rx=0 tx=0).stop 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 -- 172.21.15.115:0/1768371160 shutdown_connections 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 --2- 172.21.15.115:0/1768371160 >> v2:172.21.15.157:3300/0 conn(0x7fb4340ae8f0 0x7fb4340a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 --2- 172.21.15.115:0/1768371160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4341415c0 0x7fb4340adb60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 -- 172.21.15.115:0/1768371160 >> 172.21.15.115:0/1768371160 conn(0x7fb43406a460 msgr2=0x7fb4340630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 -- 172.21.15.115:0/1768371160 shutdown_connections 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 1 -- 172.21.15.115:0/1768371160 wait complete. 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 10 monclient: build_initial_monmap 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.461+0000 7fb43a953700 10 monclient: monmap: 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:56.476 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 librados: starting msgr at 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 librados: starting objecter 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 Processor -- start 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 -- start start 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 librados: setting wanted keys 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 librados: calling monclient init 2023-09-30T01:07:56.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 10 monclient: init 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4340ae8f0 0x7fb43413ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-a con 0x7fb4340ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.462+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-c con 0x7fb4340aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4341415c0 0x7fb43413cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): picked mon.noname-b con 0x7fb4341415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43413c2a0 con 0x7fb4340ae8f0 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43413d1b0 con 0x7fb4341415c0 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb434142940 con 0x7fb4340aecc0 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient(hunting): authenticate will time out at 1337.913037s 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4340ae8f0 0x7fb43413ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:54374/0 (socket says 172.21.15.115:54374) 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 -- 172.21.15.115:0/3438108462 learned_addr learned my addr 172.21.15.115:0/3438108462 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): get_auth_request con 0x7fb4340aecc0 auth_method 0 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:56.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): get_auth_request con 0x7fb4340ae8f0 auth_method 0 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb433fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb438ef0700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4341415c0 0x7fb43413cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient(hunting): handle_auth_done global_id 4661 payload 824 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 -- 172.21.15.115:0/3438108462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4341415c0 msgr2=0x7fb43413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4341415c0 0x7fb43413cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 -- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4340ae8f0 msgr2=0x7fb43413ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4340ae8f0 0x7fb43413ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient: _finish_hunting 0 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 monclient: found mon.noname-c 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 -- 172.21.15.115:0/3438108462 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009a20 con 0x7fb4340aecc0 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4337fe700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb42800c7e0 tx=0x7fb428007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:56.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 client.?.objecter ms_handle_connect 0x7fb4340aecc0 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428005b30 con 0x7fb4340aecc0 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: dump: 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:56.482 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: _finish_auth 0 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 5 monclient: authenticate success, global_id 4661 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb428005c90 con 0x7fb4340aecc0 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient: _renew_subs 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb434144820 con 0x7fb4340aecc0 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428007320 con 0x7fb4340aecc0 2023-09-30T01:07:56.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb41bfff700 10 monclient: dump: 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:56.484 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient: _renew_subs 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb4327fc700 4 set_mon_vals no callback set 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4341454e0 con 0x7fb4340aecc0 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.463+0000 7fb43a953700 1 librados: init done 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.464+0000 7fb43a953700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.464+0000 7fb43a953700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.464+0000 7fb43a953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.464+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43413c410 con 0x7fb4340aecc0 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb42801b3b0 con 0x7fb4340aecc0 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 1 --2- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 0x7fb42006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:56.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb433fff700 1 --2- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 0x7fb42006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb433fff700 10 monclient: get_auth_request con 0x7fb4200688c0 auth_method 0 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb4280833d0 con 0x7fb4340aecc0 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 10 client.4661.objecter ms_dispatch 0x7fb434001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 3 client.4661.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 3 client.4661.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.465+0000 7fb41bfff700 20 client.4661.objecter dump_active .. 0 homeless 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.466+0000 7fb41bfff700 10 client.4661.objecter ms_handle_connect 0x7fb4200688c0 2023-09-30T01:07:56.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.466+0000 7fb433fff700 1 --2- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 0x7fb42006ad70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb424009db0 tx=0x7fb4240091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:56.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.470+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb428050c90 con 0x7fb4340aecc0 2023-09-30T01:07:56.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.470+0000 7fb41bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:56.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.470+0000 7fb41bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:07:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb43a953700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:07:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb43a953700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb43a953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:07:56.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb434072960 con 0x7fb4340aecc0 2023-09-30T01:07:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb41bfff700 1 -- 172.21.15.115:0/3438108462 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb428017100 con 0x7fb4340aecc0 2023-09-30T01:07:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb41bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:07:56.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.694+0000 7fb41bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:07:56.695 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:07:56.695 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:56.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 10 librados: watch_flush enter 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 10 librados: watch_flush exit 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 20 client.4661.objecter shutdown clearing up homeless session... 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 10 client.4661.objecter successfully canceled tick 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 msgr2=0x7fb42006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 0x7fb42006ad70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb424009db0 tx=0x7fb4240091a0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 10 monclient: shutdown 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 msgr2=0x7fb43413c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb42800c7e0 tx=0x7fb428007df0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 shutdown_connections 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> v2:172.21.15.115:6816/115635 conn(0x7fb4200688c0 0x7fb42006ad70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4341415c0 0x7fb43413cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4340ae8f0 0x7fb43413ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 --2- 172.21.15.115:0/3438108462 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb4340aecc0 0x7fb43413c730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 >> 172.21.15.115:0/3438108462 conn(0x7fb43406a460 msgr2=0x7fb43413a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 shutdown_connections 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 -- 172.21.15.115:0/3438108462 wait complete. 2023-09-30T01:07:56.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:56.696+0000 7fb43a953700 1 librados: shutdown 2023-09-30T01:07:59.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:59.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.837+0000 7f0ab12b0700 10 monclient: get_monmap_and_config 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.837+0000 7f0ab12b0700 10 monclient: build_initial_monmap 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.837+0000 7f0ab12b0700 10 monclient: monmap: 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:07:59.838814+0000 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:07:59.838814+0000 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:07:59.838 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:07:59.839 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:07:59.839 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:07:59.839 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:07:59.839 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.838+0000 7f0ab12b0700 1 Processor -- start 2023-09-30T01:07:59.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.838+0000 7f0ab12b0700 1 -- start start 2023-09-30T01:07:59.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.838+0000 7f0ab12b0700 10 monclient: init 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-a con 0x7f0aac0632a0 addr v2:172.21.15.115:3300/0 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0aac13e320 addr v1:172.21.15.115:6789/0 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-b con 0x7f0aac06e000 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac141c60 con 0x7f0aac13e320 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0ad3f0 con 0x7f0aac0632a0 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:59.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac063670 con 0x7f0aac06e000 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient(hunting): authenticate will time out at 1341.289106s 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:54380/0 (socket says 172.21.15.115:54380) 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaa7fc700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f0aac13e320 0x7f0aac1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:57848/0 (socket says 172.21.15.115:57848) 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52182/0 (socket says 172.21.15.115:52182) 2023-09-30T01:07:59.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 -- 172.21.15.115:0/2990722665 learned_addr learned my addr 172.21.15.115:0/2990722665 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): get_auth_request con 0x7f0aac06e000 auth_method 0 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): get_auth_request con 0x7f0aac0632a0 auth_method 0 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:59.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:59.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:59.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_done global_id 4664 payload 274 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 -- 172.21.15.115:0/2990722665 >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 msgr2=0x7f0aac06f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 --2- 172.21.15.115:0/2990722665 >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 -- 172.21.15.115:0/2990722665 >> v1:172.21.15.115:6789/0 conn(0x7f0aac13e320 legacy=0x7f0aac1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 monclient: found mon.noname-b 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 -- 172.21.15.115:0/2990722665 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac13e780 con 0x7f0aac06e000 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aaaffd700 1 --2- 172.21.15.115:0/2990722665 >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:59.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aab7fe700 1 --2- 172.21.15.115:0/2990722665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009f20 tx=0x7f0aa4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4184d60469eb32f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 1 -- 172.21.15.115:0/2990722665 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400e070 con 0x7f0aac06e000 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: dump: 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:59.852 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 5 monclient: authenticate success, global_id 4664 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 1 -- 172.21.15.115:0/2990722665 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400ba60 con 0x7f0aac06e000 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 10 monclient: get_monmap_and_config success 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 1 -- 172.21.15.115:0/2990722665 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa40092e0 con 0x7f0aac06e000 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0ab12b0700 4 set_mon_vals no callback set 2023-09-30T01:07:59.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.839+0000 7f0aa9ffb700 10 monclient: dump: 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:59.854 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 10 monclient: shutdown 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2990722665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 msgr2=0x7f0aac06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2990722665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009f20 tx=0x7f0aa4009840 comp rx=0 tx=0).stop 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2990722665 shutdown_connections 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2990722665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac06e000 0x7f0aac06e460 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2990722665 >> v2:172.21.15.115:3300/0 conn(0x7f0aac0632a0 0x7f0aac06f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2990722665 >> 172.21.15.115:0/2990722665 conn(0x7f0aac0677f0 msgr2=0x7f0aac067bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2990722665 shutdown_connections 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.840+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2990722665 wait complete. 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 10 monclient: build_initial_monmap 2023-09-30T01:07:59.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 10 monclient: monmap: 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 librados: starting msgr at 2023-09-30T01:07:59.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 librados: starting objecter 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 Processor -- start 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 -- start start 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 librados: setting wanted keys 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 1 librados: calling monclient init 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.841+0000 7f0ab12b0700 10 monclient: init 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _reopen_session rank -1 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0aac0632a0 0x7f0aac0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-b con 0x7f0aac0632a0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-a con 0x7f0aac06e000 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): picked mon.noname-c con 0x7f0aac13e320 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1191d0 con 0x7f0aac06e000 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac11a2a0 con 0x7f0aac0632a0 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac11a580 con 0x7f0aac13e320 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): _renew_subs 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient(hunting): authenticate will time out at 1341.292191s 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:54382/0 (socket says 172.21.15.115:54382) 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52192/0 (socket says 172.21.15.115:52192) 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 1 -- 172.21.15.115:0/2898418942 learned_addr learned my addr 172.21.15.115:0/2898418942 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): get_auth_request con 0x7f0aac13e320 auth_method 0 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): get_auth_request con 0x7f0aac06e000 auth_method 0 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:59.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient(hunting): handle_auth_done global_id 4914 payload 824 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaaffd700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0aac0632a0 0x7f0aac0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 -- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 msgr2=0x7f0aac0af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 -- 172.21.15.115:0/2898418942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0aac0632a0 msgr2=0x7f0aac0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0aac0632a0 0x7f0aac0b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:07:59.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 monclient: found mon.noname-a 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 -- 172.21.15.115:0/2898418942 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa0009420 con 0x7f0aac06e000 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aab7fe700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aaa7fc700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c590 tx=0x7f0a9c007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 client.?.objecter ms_handle_connect 0x7f0aac06e000 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c005b40 con 0x7f0aac06e000 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: dump: 2023-09-30T01:07:59.861 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: _finish_auth 0 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c005cc0 con 0x7f0aac06e000 2023-09-30T01:07:59.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 5 monclient: authenticate success, global_id 4914 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _renew_subs 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa4009ae0 con 0x7f0aac06e000 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c007320 con 0x7f0aac06e000 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0a92ffd700 10 monclient: dump: 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.863 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0aa97fa700 4 set_mon_vals no callback set 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 client.4914.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _renew_subs 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac0b07f0 con 0x7f0aac06e000 2023-09-30T01:07:59.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.842+0000 7f0ab12b0700 1 librados: init done 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.843+0000 7f0ab12b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.843+0000 7f0ab12b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.843+0000 7f0ab12b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.843+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac072960 con 0x7f0aac06e000 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0a9c01b3b0 con 0x7f0aac06e000 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 1 --2- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 0x7f0a9806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0aaaffd700 1 --2- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 0x7f0a9806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0a9c0833e0 con 0x7f0aac06e000 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 10 client.4914.objecter ms_dispatch 0x7f0aac001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 3 client.4914.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0a92ffd700 3 client.4914.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:07:59.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.844+0000 7f0aaaffd700 10 monclient: get_auth_request con 0x7f0a98068910 auth_method 0 2023-09-30T01:07:59.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.845+0000 7f0a92ffd700 20 client.4914.objecter dump_active .. 0 homeless 2023-09-30T01:07:59.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.845+0000 7f0a92ffd700 10 client.4914.objecter ms_handle_connect 0x7f0a98068910 2023-09-30T01:07:59.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.845+0000 7f0aaaffd700 1 --2- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 0x7f0a9806adc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0006f00 tx=0x7f0aa0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:07:59.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.849+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a9c050c00 con 0x7f0aac06e000 2023-09-30T01:07:59.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.849+0000 7f0a92ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:07:59.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:07:59.849+0000 7f0a92ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0ab12b0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0ab12b0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0ab12b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0aac06a560 con 0x7f0aac06e000 2023-09-30T01:08:00.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0a92ffd700 1 -- 172.21.15.115:0/2898418942 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0a9c017100 con 0x7f0aac06e000 2023-09-30T01:08:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0a92ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.070+0000 7f0a92ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:00.071 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:00.071 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 10 librados: watch_flush enter 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 10 librados: watch_flush exit 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 20 client.4914.objecter shutdown clearing up homeless session... 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 10 client.4914.objecter successfully canceled tick 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 msgr2=0x7f0a9806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 0x7f0a9806adc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0006f00 tx=0x7f0aa0008040 comp rx=0 tx=0).stop 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 10 monclient: shutdown 2023-09-30T01:08:00.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 msgr2=0x7f0aac0aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c590 tx=0x7f0a9c007eb0 comp rx=0 tx=0).stop 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 shutdown_connections 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> v2:172.21.15.115:6816/115635 conn(0x7f0a98068910 0x7f0a9806adc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f0aac13e320 0x7f0aac0af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0aac0632a0 0x7f0aac0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 --2- 172.21.15.115:0/2898418942 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0aac06e000 0x7f0aac0aeed0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 >> 172.21.15.115:0/2898418942 conn(0x7f0aac0677f0 msgr2=0x7f0aac0abef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 shutdown_connections 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 -- 172.21.15.115:0/2898418942 wait complete. 2023-09-30T01:08:00.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:00.072+0000 7f0ab12b0700 1 librados: shutdown 2023-09-30T01:08:03.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:03.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.213+0000 7f040249f700 10 monclient: get_monmap_and_config 2023-09-30T01:08:03.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.213+0000 7f040249f700 10 monclient: build_initial_monmap 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.213+0000 7f040249f700 10 monclient: monmap: 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:03.214576+0000 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:03.214576+0000 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:03.214 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:03.215 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:03.215 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:03.215 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:03.215 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.214+0000 7f040249f700 1 Processor -- start 2023-09-30T01:08:03.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.214+0000 7f040249f700 1 -- start start 2023-09-30T01:08:03.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.214+0000 7f040249f700 10 monclient: init 2023-09-30T01:08:03.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.214+0000 7f040249f700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f03fc0ae8f0 0x7f03fc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): picked mon.noname-c con 0x7f03fc0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f03fc0aecc0 0x7f03fc0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): picked mon.noname-a con 0x7f03fc0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03fc1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc0a7040 con 0x7f03fc1415c0 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc0b19b0 con 0x7f03fc0aecc0 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:03.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc0a57f0 con 0x7f03fc0ae8f0 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient(hunting): authenticate will time out at 1344.665128s 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f03fc0aecc0 0x7f03fc0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f0400a3c700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f03fc1415c0 0x7f03fc0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:55878/0 (socket says 172.21.15.115:55878) 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f0400a3c700 1 -- 172.21.15.115:0/3422003411 learned_addr learned my addr 172.21.15.115:0/3422003411 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): get_auth_request con 0x7f03fc0aecc0 auth_method 0 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:03.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fbfff700 1 --2- 172.21.15.115:0/3422003411 >> v2:172.21.15.157:3300/0 conn(0x7f03fc0ae8f0 0x7f03fc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fbfff700 10 monclient(hunting): get_auth_request con 0x7f03fc0ae8f0 auth_method 0 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_done global_id 4917 payload 274 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 -- 172.21.15.115:0/3422003411 >> v2:172.21.15.157:3300/0 conn(0x7f03fc0ae8f0 msgr2=0x7f03fc0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 --2- 172.21.15.115:0/3422003411 >> v2:172.21.15.157:3300/0 conn(0x7f03fc0ae8f0 0x7f03fc0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 -- 172.21.15.115:0/3422003411 >> v1:172.21.15.115:6789/0 conn(0x7f03fc1415c0 legacy=0x7f03fc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:08:03.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 monclient: found mon.noname-a 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 1 -- 172.21.15.115:0/3422003411 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471643533 0 0) 0x7f03fc0a7040 con 0x7f03fc1415c0 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 -- 172.21.15.115:0/3422003411 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc0a5480 con 0x7f03fc0aecc0 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03fb7fe700 1 --2- 172.21.15.115:0/3422003411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc0aecc0 0x7f03fc0b1470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0b59c0 tx=0x7f03ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e75a5b701cf2e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 1 -- 172.21.15.115:0/3422003411 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00b6b0 con 0x7f03fc0aecc0 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: dump: 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:03.228 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: _finish_auth 0 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 1 -- 172.21.15.115:0/3422003411 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec00b810 con 0x7f03fc0aecc0 2023-09-30T01:08:03.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 5 monclient: authenticate success, global_id 4917 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 1 -- 172.21.15.115:0/3422003411 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00bbb0 con 0x7f03fc0aecc0 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f040249f700 4 set_mon_vals no callback set 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.215+0000 7f03faffd700 10 monclient: dump: 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.230 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 10 monclient: shutdown 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 -- 172.21.15.115:0/3422003411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc0aecc0 msgr2=0x7f03fc0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 --2- 172.21.15.115:0/3422003411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc0aecc0 0x7f03fc0b1470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0b59c0 tx=0x7f03ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 -- 172.21.15.115:0/3422003411 shutdown_connections 2023-09-30T01:08:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 --2- 172.21.15.115:0/3422003411 >> v2:172.21.15.157:3300/0 conn(0x7f03fc0ae8f0 0x7f03fc0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 --2- 172.21.15.115:0/3422003411 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc0aecc0 0x7f03fc0b1470 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 -- 172.21.15.115:0/3422003411 >> 172.21.15.115:0/3422003411 conn(0x7f03fc06a460 msgr2=0x7f03fc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 -- 172.21.15.115:0/3422003411 shutdown_connections 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.216+0000 7f040249f700 1 -- 172.21.15.115:0/3422003411 wait complete. 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 10 monclient: build_initial_monmap 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 10 monclient: monmap: 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:03.232 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 librados: starting msgr at 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 librados: starting objecter 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 Processor -- start 2023-09-30T01:08:03.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 -- start start 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 librados: setting wanted keys 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 1 librados: calling monclient init 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.217+0000 7f040249f700 10 monclient: init 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): picked mon.noname-c con 0x7f03fc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03fc0aecc0 0x7f03fc1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): picked mon.noname-b con 0x7f03fc0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): picked mon.noname-a con 0x7f03fc1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:03.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0aa570 con 0x7f03fc1415c0 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc151d70 con 0x7f03fc0aecc0 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0aaa00 con 0x7f03fc0ae8f0 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f040249f700 10 monclient(hunting): authenticate will time out at 1344.668255s 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54046/0 (socket says 172.21.15.115:54046) 2023-09-30T01:08:03.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47738/0 (socket says 172.21.15.115:47738) 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 -- 172.21.15.115:0/2979688919 learned_addr learned my addr 172.21.15.115:0/2979688919 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): get_auth_request con 0x7f03fc1415c0 auth_method 0 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): get_auth_request con 0x7f03fc0ae8f0 auth_method 0 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03fc0aecc0 0x7f03fc1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): get_auth_request con 0x7f03fc0aecc0 auth_method 0 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:03.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient(hunting): handle_auth_done global_id 4923 payload 824 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 -- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 msgr2=0x7f03fc13e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 -- 172.21.15.115:0/2979688919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03fc0aecc0 msgr2=0x7f03fc1512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03fc0aecc0 0x7f03fc1512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 monclient: found mon.noname-a 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 -- 172.21.15.115:0/2979688919 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec009420 con 0x7f03fc1415c0 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03fbfff700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:03.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f0400a3c700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03f400cbf0 tx=0x7f03f40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 client.?.objecter ms_handle_connect 0x7f03fc1415c0 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f400a8b0 con 0x7f03fc1415c0 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 monclient: dump: 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:03.239 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 10 monclient: _finish_auth 0 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.218+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f400aa10 con 0x7f03fc1415c0 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 5 monclient: authenticate success, global_id 4923 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03e3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f400add0 con 0x7f03fc1415c0 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _renew_subs 2023-09-30T01:08:03.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc0ad860 con 0x7f03fc1415c0 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03fa7fc700 4 set_mon_vals no callback set 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f03e3fff700 10 monclient: dump: 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:03.241 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 client.4923.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _renew_subs 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc0a95e0 con 0x7f03fc1415c0 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 1 librados: init done 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.219+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc0aa720 con 0x7f03fc1415c0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f03f4007300 con 0x7f03fc1415c0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 1 --2- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 0x7f03e806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03fbfff700 1 --2- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 0x7f03e806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f03f4084030 con 0x7f03fc1415c0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 10 client.4923.objecter ms_dispatch 0x7f03fc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 3 client.4923.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03e3fff700 3 client.4923.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.220+0000 7f03fbfff700 10 monclient: get_auth_request con 0x7f03e8068890 auth_method 0 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.221+0000 7f03e3fff700 20 client.4923.objecter dump_active .. 0 homeless 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.221+0000 7f03e3fff700 10 client.4923.objecter ms_handle_connect 0x7f03e8068890 2023-09-30T01:08:03.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.221+0000 7f03fbfff700 1 --2- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 0x7f03e806ad40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03f0006fd0 tx=0x7f03f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:03.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.225+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03fc0aa720 con 0x7f03fc1415c0 2023-09-30T01:08:03.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.225+0000 7f03e3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:03.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.225+0000 7f03e3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.445+0000 7f040249f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.445+0000 7f040249f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.445+0000 7f040249f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.445+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03fc0a95e0 con 0x7f03fc1415c0 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.446+0000 7f03e3fff700 1 -- 172.21.15.115:0/2979688919 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f03fc0a95e0 con 0x7f03fc1415c0 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.446+0000 7f03e3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:03.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.446+0000 7f03e3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:03.447 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.447+0000 7f040249f700 10 librados: watch_flush enter 2023-09-30T01:08:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.447+0000 7f040249f700 10 librados: watch_flush exit 2023-09-30T01:08:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.447+0000 7f040249f700 20 client.4923.objecter shutdown clearing up homeless session... 2023-09-30T01:08:03.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.447+0000 7f040249f700 10 client.4923.objecter successfully canceled tick 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 msgr2=0x7f03e806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 0x7f03e806ad40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03f0006fd0 tx=0x7f03f0008040 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 10 monclient: shutdown 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 msgr2=0x7f03fc151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03f400cbf0 tx=0x7f03f40019b0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 shutdown_connections 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> v2:172.21.15.115:6816/115635 conn(0x7f03e8068890 0x7f03e806ad40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f03fc1415c0 0x7f03fc151830 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f03fc0ae8f0 0x7f03fc13e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 --2- 172.21.15.115:0/2979688919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03fc0aecc0 0x7f03fc1512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:03.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 >> 172.21.15.115:0/2979688919 conn(0x7f03fc06a460 msgr2=0x7f03fc144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 shutdown_connections 2023-09-30T01:08:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 -- 172.21.15.115:0/2979688919 wait complete. 2023-09-30T01:08:03.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:03.448+0000 7f040249f700 1 librados: shutdown 2023-09-30T01:08:06.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.590+0000 7f6f9eba0700 10 monclient: get_monmap_and_config 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.590+0000 7f6f9eba0700 10 monclient: build_initial_monmap 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.590+0000 7f6f9eba0700 10 monclient: monmap: 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:06.591043+0000 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:06.591043+0000 2023-09-30T01:08:06.590 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:06.591 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.590+0000 7f6f9eba0700 1 Processor -- start 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 -- start start 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient: init 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f980ae8f0 0x7f6f980a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-c con 0x7f6f980ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f980aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-b con 0x7f6f981415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980a7040 con 0x7f6f980aecc0 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f980a57f0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980a5480 con 0x7f6f981415c0 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9eba0700 10 monclient(hunting): authenticate will time out at 1348.041663s 2023-09-30T01:08:06.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47754/0 (socket says 172.21.15.115:47754) 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 1 -- 172.21.15.115:0/62801185 learned_addr learned my addr 172.21.15.115:0/62801185 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): get_auth_request con 0x7f6f981415c0 auth_method 0 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.591+0000 7f6f9c93c700 1 --2- 172.21.15.115:0/62801185 >> v2:172.21.15.157:3300/0 conn(0x7f6f980ae8f0 0x7f6f980a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): get_auth_request con 0x7f6f980ae8f0 auth_method 0 2023-09-30T01:08:06.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_done global_id 4673 payload 274 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 -- 172.21.15.115:0/62801185 >> v2:172.21.15.157:3300/0 conn(0x7f6f980ae8f0 msgr2=0x7f6f980a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 --2- 172.21.15.115:0/62801185 >> v2:172.21.15.157:3300/0 conn(0x7f6f980ae8f0 0x7f6f980a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 -- 172.21.15.115:0/62801185 >> v1:172.21.15.157:6789/0 conn(0x7f6f980aecc0 legacy=0x7f6f980b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:06.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 monclient: found mon.noname-b 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 -- 172.21.15.115:0/62801185 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f98141a40 con 0x7f6f981415c0 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9d13d700 1 --2- 172.21.15.115:0/62801185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009f20 tx=0x7f6f90009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d81f1a34e36a207f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 1 -- 172.21.15.115:0/62801185 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9000e070 con 0x7f6f981415c0 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: dump: 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.604 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9eba0700 5 monclient: authenticate success, global_id 4673 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9eba0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 1 -- 172.21.15.115:0/62801185 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9000b9d0 con 0x7f6f981415c0 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:06.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9eba0700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 1 -- 172.21.15.115:0/62801185 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f900092e0 con 0x7f6f981415c0 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f9eba0700 4 set_mon_vals no callback set 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.592+0000 7f6f977fe700 10 monclient: dump: 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:06.606 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 10 monclient: shutdown 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 -- 172.21.15.115:0/62801185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 msgr2=0x7f6f980adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/62801185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009f20 tx=0x7f6f90009840 comp rx=0 tx=0).stop 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 -- 172.21.15.115:0/62801185 shutdown_connections 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/62801185 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f981415c0 0x7f6f980adb60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/62801185 >> v2:172.21.15.157:3300/0 conn(0x7f6f980ae8f0 0x7f6f980a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 -- 172.21.15.115:0/62801185 >> 172.21.15.115:0/62801185 conn(0x7f6f9806a460 msgr2=0x7f6f980630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 -- 172.21.15.115:0/62801185 shutdown_connections 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 -- 172.21.15.115:0/62801185 wait complete. 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 10 monclient: build_initial_monmap 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 10 monclient: monmap: 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:06.608 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 librados: starting msgr at 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.593+0000 7f6f9eba0700 1 librados: starting objecter 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 Processor -- start 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 -- start start 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 librados: setting wanted keys 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 librados: calling monclient init 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient: init 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:06.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-c con 0x7f6f980ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-a con 0x7f6f980aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f981415c0 0x7f6f980abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): picked mon.noname-b con 0x7f6f981415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980ab5d0 con 0x7f6f980aecc0 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980ab8b0 con 0x7f6f981415c0 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:06.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980a8460 con 0x7f6f980ae8f0 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9eba0700 10 monclient(hunting): authenticate will time out at 1348.044857s 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47766/0 (socket says 172.21.15.115:47766) 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54052/0 (socket says 172.21.15.115:54052) 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 1 -- 172.21.15.115:0/1869490970 learned_addr learned my addr 172.21.15.115:0/1869490970 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 10 monclient(hunting): get_auth_request con 0x7f6f980ae8f0 auth_method 0 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f9c93c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:06.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 10 monclient(hunting): get_auth_request con 0x7f6f980aecc0 auth_method 0 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.594+0000 7f6f97fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f981415c0 0x7f6f980abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): get_auth_request con 0x7f6f981415c0 auth_method 0 2023-09-30T01:08:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9d13d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient(hunting): handle_auth_done global_id 4676 payload 824 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 -- 172.21.15.115:0/1869490970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f981415c0 msgr2=0x7f6f980abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f981415c0 0x7f6f980abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 -- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 msgr2=0x7f6f980a7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 monclient: found mon.noname-c 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 -- 172.21.15.115:0/1869490970 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90009ae0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f97fff700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9c93c700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00ec80 tx=0x7f6f8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 client.?.objecter ms_handle_connect 0x7f6f980ae8f0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00c850 con 0x7f6f980ae8f0 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:06.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: dump: 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:06.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c00c9b0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 5 monclient: authenticate success, global_id 4676 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00cd50 con 0x7f6f980ae8f0 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 10 monclient: _renew_subs 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f980a8160 con 0x7f6f980ae8f0 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f96ffd700 4 set_mon_vals no callback set 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f94ff9700 10 monclient: dump: 2023-09-30T01:08:06.616 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 10 monclient: _renew_subs 2023-09-30T01:08:06.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f980ac5e0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.595+0000 7f6f9eba0700 1 librados: init done 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.596+0000 7f6f9eba0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.596+0000 7f6f9eba0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.596+0000 7f6f9eba0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.596+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f980a8680 con 0x7f6f980ae8f0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.596+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6f8c013680 con 0x7f6f980ae8f0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 1 --2- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 0x7f6f7c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6f8c084100 con 0x7f6f980ae8f0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f97fff700 1 --2- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 0x7f6f7c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 10 client.4676.objecter ms_dispatch 0x7f6f98001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:06.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 3 client.4676.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 3 client.4676.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f97fff700 10 monclient: get_auth_request con 0x7f6f7c068890 auth_method 0 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 20 client.4676.objecter dump_active .. 0 homeless 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f94ff9700 10 client.4676.objecter ms_handle_connect 0x7f6f7c068890 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.597+0000 7f6f97fff700 1 --2- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 0x7f6f7c06ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f98063250 tx=0x7f6f84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:06.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.601+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f8c0519c0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.601+0000 7f6f94ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:06.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.601+0000 7f6f94ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.824+0000 7f6f9eba0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.824+0000 7f6f9eba0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.824+0000 7f6f9eba0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.824+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f98072960 con 0x7f6f980ae8f0 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.825+0000 7f6f94ff9700 1 -- 172.21.15.115:0/1869490970 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6f8c054fe0 con 0x7f6f980ae8f0 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.825+0000 7f6f94ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.825+0000 7f6f94ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:06.825 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:06.826 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 10 librados: watch_flush enter 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 10 librados: watch_flush exit 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 20 client.4676.objecter shutdown clearing up homeless session... 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 10 client.4676.objecter successfully canceled tick 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 msgr2=0x7f6f7c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 0x7f6f7c06ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f98063250 tx=0x7f6f84009040 comp rx=0 tx=0).stop 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.826+0000 7f6f9eba0700 10 monclient: shutdown 2023-09-30T01:08:06.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 msgr2=0x7f6f9813e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00ec80 tx=0x7f6f8c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 shutdown_connections 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> v2:172.21.15.115:6816/115635 conn(0x7f6f7c068890 0x7f6f7c06ad40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f981415c0 0x7f6f980abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6f980ae8f0 0x7f6f9813e450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 --2- 172.21.15.115:0/1869490970 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f980aecc0 0x7f6f980a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 >> 172.21.15.115:0/1869490970 conn(0x7f6f9806a460 msgr2=0x7f6f9813fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 shutdown_connections 2023-09-30T01:08:06.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 -- 172.21.15.115:0/1869490970 wait complete. 2023-09-30T01:08:06.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:06.827+0000 7f6f9eba0700 1 librados: shutdown 2023-09-30T01:08:09.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:09.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.967+0000 7efef0e62700 10 monclient: get_monmap_and_config 2023-09-30T01:08:09.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.967+0000 7efef0e62700 10 monclient: build_initial_monmap 2023-09-30T01:08:09.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.967+0000 7efef0e62700 10 monclient: monmap: 2023-09-30T01:08:09.967 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:09.968361+0000 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:09.968361+0000 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:09.968 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:09.969 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.967+0000 7efef0e62700 1 Processor -- start 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 -- start start 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient: init 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T01:08:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efeec0ae8f0 0x7efeec0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-c con 0x7efeec0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-a con 0x7efeec0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-b con 0x7efeec1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0a7040 con 0x7efeec0aecc0 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0b19b0 con 0x7efeec0ae8f0 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0a57f0 con 0x7efeec1415c0 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efef0e62700 10 monclient(hunting): authenticate will time out at 1351.418831s 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efee9d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.968+0000 7efeead9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54062/0 (socket says 172.21.15.115:54062) 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47774/0 (socket says 172.21.15.115:47774) 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 1 -- 172.21.15.115:0/1923172841 learned_addr learned my addr 172.21.15.115:0/1923172841 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): get_auth_request con 0x7efeec0aecc0 auth_method 0 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): get_auth_request con 0x7efeec1415c0 auth_method 0 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeea59c700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.157:3300/0 conn(0x7efeec0ae8f0 0x7efeec0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeea59c700 10 monclient(hunting): get_auth_request con 0x7efeec0ae8f0 auth_method 0 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeea59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeea59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient(hunting): handle_auth_done global_id 4679 payload 274 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 -- 172.21.15.115:0/1923172841 >> v2:172.21.15.157:3300/0 conn(0x7efeec0ae8f0 msgr2=0x7efeec0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.157:3300/0 conn(0x7efeec0ae8f0 0x7efeec0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 -- 172.21.15.115:0/1923172841 >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 msgr2=0x7efeec0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 monclient: found mon.noname-b 2023-09-30T01:08:09.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 -- 172.21.15.115:0/1923172841 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0a5480 con 0x7efeec1415c0 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee9d9b700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efeead9d700 1 --2- 172.21.15.115:0/1923172841 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efeec0b59c0 tx=0x7efee4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe53f5987b0c6fb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 1 -- 172.21.15.115:0/1923172841 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee400e070 con 0x7efeec1415c0 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: dump: 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:09.982 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: _finish_auth 0 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 1 -- 172.21.15.115:0/1923172841 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee400bb60 con 0x7efeec1415c0 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efef0e62700 5 monclient: authenticate success, global_id 4679 2023-09-30T01:08:09.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efef0e62700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 1 -- 172.21.15.115:0/1923172841 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4012430 con 0x7efeec1415c0 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efee959a700 10 monclient: dump: 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efef0e62700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.969+0000 7efef0e62700 4 set_mon_vals no callback set 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 10 monclient: shutdown 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:09.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 -- 172.21.15.115:0/1923172841 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 msgr2=0x7efeec0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 --2- 172.21.15.115:0/1923172841 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efeec0b59c0 tx=0x7efee4009780 comp rx=0 tx=0).stop 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 -- 172.21.15.115:0/1923172841 shutdown_connections 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 --2- 172.21.15.115:0/1923172841 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec1415c0 0x7efeec0adb60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.157:3300/0 conn(0x7efeec0ae8f0 0x7efeec0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 --2- 172.21.15.115:0/1923172841 >> v2:172.21.15.115:3300/0 conn(0x7efeec0aecc0 0x7efeec0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 -- 172.21.15.115:0/1923172841 >> 172.21.15.115:0/1923172841 conn(0x7efeec06a460 msgr2=0x7efeec0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 -- 172.21.15.115:0/1923172841 shutdown_connections 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 1 -- 172.21.15.115:0/1923172841 wait complete. 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 10 monclient: build_initial_monmap 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.970+0000 7efef0e62700 10 monclient: monmap: 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:09.986 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 librados: starting msgr at 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 librados: starting objecter 2023-09-30T01:08:09.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 Processor -- start 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 -- start start 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 librados: setting wanted keys 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 librados: calling monclient init 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient: init 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-c con 0x7efeec0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 0x7efeec1513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-b con 0x7efeec0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 0x7efeec0abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): picked mon.noname-a con 0x7efeec1415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec151920 con 0x7efeec1415c0 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0aba00 con 0x7efeec0aecc0 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec151b80 con 0x7efeec0ae8f0 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efef0e62700 10 monclient(hunting): authenticate will time out at 1351.421799s 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:47784/0 (socket says 172.21.15.115:47784) 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 0x7efeec0abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 1 -- 172.21.15.115:0/4066746580 learned_addr learned my addr 172.21.15.115:0/4066746580 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): get_auth_request con 0x7efeec0ae8f0 auth_method 0 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): get_auth_request con 0x7efeec1415c0 auth_method 0 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:09.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeea59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.971+0000 7efeead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee9d9b700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 0x7efeec1513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 10 monclient(hunting): handle_auth_done global_id 4932 payload 824 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 -- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 msgr2=0x7efeec13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 -- 172.21.15.115:0/4066746580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 msgr2=0x7efeec1513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 0x7efeec1513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:09.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 monclient: found mon.noname-a 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 -- 172.21.15.115:0/4066746580 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc0094f0 con 0x7efeec1415c0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee9d9b700 10 monclient: get_auth_request con 0x7efeec0aecc0 auth_method 0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee9d9b700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 0x7efeec1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeea59c700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efeead9d700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 0x7efeec0abfb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efee4016000 tx=0x7efee4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 client.?.objecter ms_handle_connect 0x7efeec1415c0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4027470 con 0x7efeec1415c0 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:09.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: dump: 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 5 monclient: authenticate success, global_id 4932 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee40275f0 con 0x7efeec1415c0 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 10 monclient: _renew_subs 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee4009a20 con 0x7efeec1415c0 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40279b0 con 0x7efeec1415c0 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee27fc700 10 monclient: dump: 2023-09-30T01:08:09.994 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efee8d99700 4 set_mon_vals no callback set 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 10 client.4932.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:09.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 10 monclient: _renew_subs 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec0ac570 con 0x7efeec1415c0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.972+0000 7efef0e62700 1 librados: init done 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.973+0000 7efef0e62700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.973+0000 7efef0e62700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.973+0000 7efef0e62700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.973+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec151da0 con 0x7efeec1415c0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.973+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efee4024070 con 0x7efeec1415c0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 1 --2- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 0x7efecc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efee40865a0 con 0x7efeec1415c0 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 10 client.4932.objecter ms_dispatch 0x7efeec001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:09.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 3 client.4932.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 3 client.4932.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efeea59c700 1 --2- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 0x7efecc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efeea59c700 10 monclient: get_auth_request con 0x7efecc068890 auth_method 0 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 20 client.4932.objecter dump_active .. 0 homeless 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efee27fc700 10 client.4932.objecter ms_handle_connect 0x7efecc068890 2023-09-30T01:08:09.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.974+0000 7efeea59c700 1 --2- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 0x7efecc06ad40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efed4006fd0 tx=0x7efed4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:09.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.978+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efee4053e60 con 0x7efeec1415c0 2023-09-30T01:08:09.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.978+0000 7efee27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:09.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:09.978+0000 7efee27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efef0e62700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:10.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efef0e62700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efef0e62700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efeec072960 con 0x7efeec1415c0 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efee27fc700 1 -- 172.21.15.115:0/4066746580 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efee4009880 con 0x7efeec1415c0 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efee27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.202+0000 7efee27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:10.203 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:10.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 10 librados: watch_flush enter 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 10 librados: watch_flush exit 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 20 client.4932.objecter shutdown clearing up homeless session... 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 10 client.4932.objecter successfully canceled tick 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 msgr2=0x7efecc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 0x7efecc06ad40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efed4006fd0 tx=0x7efed4008040 comp rx=0 tx=0).stop 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 10 monclient: shutdown 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 msgr2=0x7efeec0abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 0x7efeec0abfb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efee4016000 tx=0x7efee4019000 comp rx=0 tx=0).stop 2023-09-30T01:08:10.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 shutdown_connections 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> v2:172.21.15.115:6816/115635 conn(0x7efecc068890 0x7efecc06ad40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efeec1415c0 0x7efeec0abfb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efeec0ae8f0 0x7efeec13e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 --2- 172.21.15.115:0/4066746580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efeec0aecc0 0x7efeec1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.204+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 >> 172.21.15.115:0/4066746580 conn(0x7efeec06a460 msgr2=0x7efeec144710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.205+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 shutdown_connections 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.205+0000 7efef0e62700 1 -- 172.21.15.115:0/4066746580 wait complete. 2023-09-30T01:08:10.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:10.205+0000 7efef0e62700 1 librados: shutdown 2023-09-30T01:08:13.217 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 10 monclient: get_monmap_and_config 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 10 monclient: build_initial_monmap 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 10 monclient: monmap: 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:13.348081+0000 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:13.348081+0000 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:13.347 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:13.348 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 1 Processor -- start 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 1 -- start start 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.347+0000 7fd3b8a3a700 10 monclient: init 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3b40ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd3b40aecc0 0x7fd3b40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-a con 0x7fd3b40aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd3b41415c0 0x7fd3b40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-c con 0x7fd3b41415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40a7040 con 0x7fd3b40ae8f0 2023-09-30T01:08:13.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40a57f0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40a5480 con 0x7fd3b41415c0 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b8a3a700 10 monclient(hunting): authenticate will time out at 1354.798489s 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd3b40aecc0 0x7fd3b40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b259c700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd3b40ae8f0 0x7fd3b40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:35346/0 (socket says 172.21.15.115:35346) 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b259c700 1 -- 172.21.15.115:0/2699720336 learned_addr learned my addr 172.21.15.115:0/2699720336 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fd3b40aecc0 0x7fd3b40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40082/0 (socket says 172.21.15.115:40082) 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): get_auth_request con 0x7fd3b40aecc0 auth_method 0 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:13.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b2d9d700 1 --2- 172.21.15.115:0/2699720336 >> v2:172.21.15.157:3300/0 conn(0x7fd3b41415c0 0x7fd3b40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b2d9d700 10 monclient(hunting): get_auth_request con 0x7fd3b41415c0 auth_method 0 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_done global_id 4935 payload 274 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/2699720336 >> v2:172.21.15.157:3300/0 conn(0x7fd3b41415c0 msgr2=0x7fd3b40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:13.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 --2- 172.21.15.115:0/2699720336 >> v2:172.21.15.157:3300/0 conn(0x7fd3b41415c0 0x7fd3b40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/2699720336 >> v1:172.21.15.115:6789/0 conn(0x7fd3b40ae8f0 legacy=0x7fd3b40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 monclient: found mon.noname-a 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/2699720336 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40a55c0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b159a700 1 -- 172.21.15.115:0/2699720336 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892501220 0 0) 0x7fd3b40a7040 con 0x7fd3b40ae8f0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.348+0000 7fd3b159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b1d9b700 1 --2- 172.21.15.115:0/2699720336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a5d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40a6250 tx=0x7fd3a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aded82f5d0ae7cb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 1 -- 172.21.15.115:0/2699720336 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800b6b0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:13.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: dump: 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: _finish_auth 0 2023-09-30T01:08:13.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 5 monclient: authenticate success, global_id 4935 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 1 -- 172.21.15.115:0/2699720336 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a800b810 con 0x7fd3b40aecc0 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 4 set_mon_vals no callback set 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 1 -- 172.21.15.115:0/2699720336 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800bbb0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b159a700 10 monclient: dump: 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:13.361 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 10 monclient: shutdown 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/2699720336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 msgr2=0x7fd3b40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/2699720336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a5d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40a6250 tx=0x7fd3a80091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/2699720336 shutdown_connections 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/2699720336 >> v2:172.21.15.157:3300/0 conn(0x7fd3b41415c0 0x7fd3b40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/2699720336 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a5d10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/2699720336 >> 172.21.15.115:0/2699720336 conn(0x7fd3b406a460 msgr2=0x7fd3b40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/2699720336 shutdown_connections 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.349+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/2699720336 wait complete. 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 10 monclient: build_initial_monmap 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 10 monclient: monmap: 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:13.363 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 librados: starting msgr at 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 librados: starting objecter 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 Processor -- start 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 -- start start 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 librados: setting wanted keys 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 1 librados: calling monclient init 2023-09-30T01:08:13.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.350+0000 7fd3b8a3a700 10 monclient: init 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3b40ae8f0 0x7fd3b413e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-b con 0x7fd3b40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-a con 0x7fd3b40aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd3b41415c0 0x7fd3b40abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): picked mon.noname-c con 0x7fd3b41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40ab920 con 0x7fd3b40aecc0 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:13.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40a85b0 con 0x7fd3b40ae8f0 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b413eee0 con 0x7fd3b41415c0 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b8a3a700 10 monclient(hunting): authenticate will time out at 1354.801363s 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40094/0 (socket says 172.21.15.115:40094) 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/1736353748 learned_addr learned my addr 172.21.15.115:0/1736353748 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd3b41415c0 0x7fd3b40abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): get_auth_request con 0x7fd3b40aecc0 auth_method 0 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:13.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): get_auth_request con 0x7fd3b41415c0 auth_method 0 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3b40ae8f0 0x7fd3b413e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): get_auth_request con 0x7fd3b40ae8f0 auth_method 0 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:13.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient(hunting): handle_auth_done global_id 4941 payload 824 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd3b41415c0 msgr2=0x7fd3b40abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd3b41415c0 0x7fd3b40abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/1736353748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3b40ae8f0 msgr2=0x7fd3b413e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3b40ae8f0 0x7fd3b413e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:08:13.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 monclient: found mon.noname-a 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 -- 172.21.15.115:0/1736353748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a8009420 con 0x7fd3b40aecc0 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3b1d9b700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40b0920 tx=0x7fd3a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3a27fc700 10 client.?.objecter ms_handle_connect 0x7fd3b40aecc0 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.351+0000 7fd3a27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800bb90 con 0x7fd3b40aecc0 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: dump: 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:13.369 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:13.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a8011420 con 0x7fd3b40aecc0 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 5 monclient: authenticate success, global_id 4941 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 monclient: _renew_subs 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b40a81c0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a80117c0 con 0x7fd3b40aecc0 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3a27fc700 10 monclient: dump: 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:13.371 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b0d99700 4 set_mon_vals no callback set 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 client.4941.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 monclient: _renew_subs 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b40ac610 con 0x7fd3b40aecc0 2023-09-30T01:08:13.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 1 librados: init done 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.352+0000 7fd3b8a3a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3b8a3a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3b8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4072960 con 0x7fd3b40aecc0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd3a8011b70 con 0x7fd3b40aecc0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 1 --2- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 0x7fd39406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd3a8085670 con 0x7fd3b40aecc0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 10 client.4941.objecter ms_dispatch 0x7fd3b4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 3 client.4941.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 3 client.4941.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3b259c700 1 --2- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 0x7fd39406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:13.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3b259c700 10 monclient: get_auth_request con 0x7fd3940688c0 auth_method 0 2023-09-30T01:08:13.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.353+0000 7fd3a27fc700 20 client.4941.objecter dump_active .. 0 homeless 2023-09-30T01:08:13.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.354+0000 7fd3a27fc700 10 client.4941.objecter ms_handle_connect 0x7fd3940688c0 2023-09-30T01:08:13.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.354+0000 7fd3b259c700 1 --2- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 0x7fd39406ad70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007fd0 tx=0x7fd3a4007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:13.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.358+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3a8052f30 con 0x7fd3b40aecc0 2023-09-30T01:08:13.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.358+0000 7fd3a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:13.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.358+0000 7fd3a27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.580+0000 7fd3b8a3a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.580+0000 7fd3b8a3a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.580+0000 7fd3b8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.580+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd3b4141a40 con 0x7fd3b40aecc0 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.581+0000 7fd3a27fc700 1 -- 172.21.15.115:0/1736353748 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd3a8056550 con 0x7fd3b40aecc0 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.581+0000 7fd3a27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:13.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.581+0000 7fd3a27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:13.582 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:13.582 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 10 librados: watch_flush enter 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 10 librados: watch_flush exit 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 20 client.4941.objecter shutdown clearing up homeless session... 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 10 client.4941.objecter successfully canceled tick 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 msgr2=0x7fd39406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 0x7fd39406ad70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007fd0 tx=0x7fd3a4007d30 comp rx=0 tx=0).stop 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 10 monclient: shutdown 2023-09-30T01:08:13.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 msgr2=0x7fd3b40a7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40b0920 tx=0x7fd3a8017000 comp rx=0 tx=0).stop 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 shutdown_connections 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> v2:172.21.15.115:6816/115635 conn(0x7fd3940688c0 0x7fd39406ad70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd3b41415c0 0x7fd3b40abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3b40ae8f0 0x7fd3b413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 --2- 172.21.15.115:0/1736353748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3b40aecc0 0x7fd3b40a7c80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 >> 172.21.15.115:0/1736353748 conn(0x7fd3b406a460 msgr2=0x7fd3b40b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 shutdown_connections 2023-09-30T01:08:13.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 -- 172.21.15.115:0/1736353748 wait complete. 2023-09-30T01:08:13.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:13.583+0000 7fd3b8a3a700 1 librados: shutdown 2023-09-30T01:08:16.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:16.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.724+0000 7ff3084c2700 10 monclient: get_monmap_and_config 2023-09-30T01:08:16.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.724+0000 7ff3084c2700 10 monclient: build_initial_monmap 2023-09-30T01:08:16.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.724+0000 7ff3084c2700 10 monclient: monmap: 2023-09-30T01:08:16.734 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:16.734 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:16.725674+0000 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:16.725674+0000 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:16.735 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.725+0000 7ff3084c2700 1 Processor -- start 2023-09-30T01:08:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.725+0000 7ff3084c2700 1 -- start start 2023-09-30T01:08:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.725+0000 7ff3084c2700 10 monclient: init 2023-09-30T01:08:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.725+0000 7ff3084c2700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:16.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3000ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff3000aecc0 0x7ff3000a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-a con 0x7ff3000aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 0x7ff3000adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-c con 0x7ff3001415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3000a7040 con 0x7ff3000ae8f0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3000a57f0 con 0x7ff3000aecc0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3000a5480 con 0x7ff3001415c0 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:16.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient(hunting): authenticate will time out at 1358.176088s 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff3000aecc0 0x7ff3000a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff3000aecc0 0x7ff3000a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40096/0 (socket says 172.21.15.115:40096) 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30625e700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff3000ae8f0 0x7ff3000b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:35350/0 (socket says 172.21.15.115:35350) 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 -- 172.21.15.115:0/3724176475 learned_addr learned my addr 172.21.15.115:0/3724176475 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): get_auth_request con 0x7ff3000aecc0 auth_method 0 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:16.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 1 --2- 172.21.15.115:0/3724176475 >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 0x7ff3000adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 10 monclient(hunting): get_auth_request con 0x7ff3001415c0 auth_method 0 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient(hunting): handle_auth_done global_id 4944 payload 274 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 -- 172.21.15.115:0/3724176475 >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 msgr2=0x7ff3000adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 --2- 172.21.15.115:0/3724176475 >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 0x7ff3000adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 -- 172.21.15.115:0/3724176475 >> v1:172.21.15.115:6789/0 conn(0x7ff3000ae8f0 legacy=0x7ff3000b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 monclient: found mon.noname-a 2023-09-30T01:08:16.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 1 -- 172.21.15.115:0/3724176475 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330755073 0 0) 0x7ff3000a7040 con 0x7ff3000ae8f0 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 -- 172.21.15.115:0/3724176475 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3000a55c0 con 0x7ff3000aecc0 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff306a5f700 1 --2- 172.21.15.115:0/3724176475 >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 0x7ff3000adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff305a5d700 1 --2- 172.21.15.115:0/3724176475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000a5d10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3000a6250 tx=0x7ff2f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9db346bc533eb96 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 1 -- 172.21.15.115:0/3724176475 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000b860 con 0x7ff3000aecc0 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: dump: 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:16.751 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: _finish_auth 0 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 5 monclient: authenticate success, global_id 4944 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 1 -- 172.21.15.115:0/3724176475 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f000b9c0 con 0x7ff3000aecc0 2023-09-30T01:08:16.752 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 1 -- 172.21.15.115:0/3724176475 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000bd60 con 0x7ff3000aecc0 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff3084c2700 4 set_mon_vals no callback set 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.726+0000 7ff30525c700 10 monclient: dump: 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:16.753 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 10 monclient: shutdown 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 -- 172.21.15.115:0/3724176475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 msgr2=0x7ff3000a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3724176475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000a5d10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3000a6250 tx=0x7ff2f0009250 comp rx=0 tx=0).stop 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 -- 172.21.15.115:0/3724176475 shutdown_connections 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3724176475 >> v2:172.21.15.157:3300/0 conn(0x7ff3001415c0 0x7ff3000adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3724176475 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000a5d10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 -- 172.21.15.115:0/3724176475 >> 172.21.15.115:0/3724176475 conn(0x7ff30006a460 msgr2=0x7ff3000630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 -- 172.21.15.115:0/3724176475 shutdown_connections 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.727+0000 7ff3084c2700 1 -- 172.21.15.115:0/3724176475 wait complete. 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient: build_initial_monmap 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient: monmap: 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 librados: starting msgr at 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 librados: starting objecter 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 Processor -- start 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 -- start start 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 librados: setting wanted keys 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 librados: calling monclient init 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient: init 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:16.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 0x7ff30013a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-c con 0x7ff3000ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-a con 0x7ff3000aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.728+0000 7ff3084c2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3001415c0 0x7ff300151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): picked mon.noname-b con 0x7ff3001415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000b4b40 con 0x7ff3000aecc0 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3000b4800 con 0x7ff3001415c0 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30013c530 con 0x7ff3000ae8f0 2023-09-30T01:08:16.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient(hunting): authenticate will time out at 1358.179027s 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 0x7ff30013a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 0x7ff30013a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50724/0 (socket says 172.21.15.115:50724) 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40102/0 (socket says 172.21.15.115:40102) 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 1 -- 172.21.15.115:0/3752695225 learned_addr learned my addr 172.21.15.115:0/3752695225 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): get_auth_request con 0x7ff3000ae8f0 auth_method 0 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:16.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): get_auth_request con 0x7ff3000aecc0 auth_method 0 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff30625e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff306a5f700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3001415c0 0x7ff300151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff306a5f700 10 monclient(hunting): get_auth_request con 0x7ff3001415c0 auth_method 0 2023-09-30T01:08:16.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff306a5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff306a5f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff306a5f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient(hunting): handle_auth_done global_id 4950 payload 824 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 -- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 msgr2=0x7ff30013a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 0x7ff30013a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 -- 172.21.15.115:0/3752695225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3001415c0 msgr2=0x7ff300151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3001415c0 0x7ff300151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:16.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 monclient: found mon.noname-a 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 -- 172.21.15.115:0/3752695225 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00094f0 con 0x7ff3000aecc0 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff305a5d700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff3000b59c0 tx=0x7ff2f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 client.?.objecter ms_handle_connect 0x7ff3000aecc0 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0026480 con 0x7ff3000aecc0 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: dump: 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:16.761 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:16.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f001f070 con 0x7ff3000aecc0 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 5 monclient: authenticate success, global_id 4950 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient: _renew_subs 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30013c700 con 0x7ff3000aecc0 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0026830 con 0x7ff3000aecc0 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff2f67fc700 10 monclient: dump: 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.763 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff304a5b700 4 set_mon_vals no callback set 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 client.4950.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient: _renew_subs 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff300151910 con 0x7ff3000aecc0 2023-09-30T01:08:16.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.729+0000 7ff3084c2700 1 librados: init done 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.730+0000 7ff3084c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.730+0000 7ff3084c2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.730+0000 7ff3084c2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.730+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3000b4cb0 con 0x7ff3000aecc0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff2f0026ba0 con 0x7ff3000aecc0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 1 --2- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 0x7ff2e406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff2f0021070 con 0x7ff3000aecc0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff30625e700 1 --2- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 0x7ff2e406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 10 client.4950.objecter ms_dispatch 0x7ff300001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 3 client.4950.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 3 client.4950.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:16.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff30625e700 10 monclient: get_auth_request con 0x7ff2e4068890 auth_method 0 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 20 client.4950.objecter dump_active .. 0 homeless 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff30625e700 1 --2- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 0x7ff2e406ad40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc007fd0 tx=0x7ff2fc007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.731+0000 7ff2f67fc700 10 client.4950.objecter ms_handle_connect 0x7ff2e4068890 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.735+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2f0053670 con 0x7ff3000aecc0 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.735+0000 7ff2f67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:16.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.735+0000 7ff2f67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.958+0000 7ff3084c2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.958+0000 7ff3084c2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.958+0000 7ff3084c2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.958+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff300072960 con 0x7ff3000aecc0 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.959+0000 7ff2f67fc700 1 -- 172.21.15.115:0/3752695225 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff2f0015080 con 0x7ff3000aecc0 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.959+0000 7ff2f67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.959+0000 7ff2f67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:16.960 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:16.960 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 10 librados: watch_flush enter 2023-09-30T01:08:16.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 10 librados: watch_flush exit 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 20 client.4950.objecter shutdown clearing up homeless session... 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 10 client.4950.objecter successfully canceled tick 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 msgr2=0x7ff2e406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 0x7ff2e406ad40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc007fd0 tx=0x7ff2fc007e10 comp rx=0 tx=0).stop 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 10 monclient: shutdown 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 msgr2=0x7ff3000b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff3000b59c0 tx=0x7ff2f0017000 comp rx=0 tx=0).stop 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 shutdown_connections 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> v2:172.21.15.115:6816/115635 conn(0x7ff2e4068890 0x7ff2e406ad40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3001415c0 0x7ff300151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff3000ae8f0 0x7ff30013a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 --2- 172.21.15.115:0/3752695225 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3000aecc0 0x7ff3000b42c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 >> 172.21.15.115:0/3752695225 conn(0x7ff30006a460 msgr2=0x7ff3000b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 shutdown_connections 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 -- 172.21.15.115:0/3752695225 wait complete. 2023-09-30T01:08:16.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:16.961+0000 7ff3084c2700 1 librados: shutdown 2023-09-30T01:08:19.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.102+0000 7fe9c2162700 10 monclient: get_monmap_and_config 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.102+0000 7fe9c2162700 10 monclient: build_initial_monmap 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.102+0000 7fe9c2162700 10 monclient: monmap: 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:20.102 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:20.103281+0000 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:20.103281+0000 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:20.103 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 Processor -- start 2023-09-30T01:08:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 -- start start 2023-09-30T01:08:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient: init 2023-09-30T01:08:20.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe9bc0ae8f0 0x7fe9bc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9bc1415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0a7040 con 0x7fe9bc1415c0 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9bc0b19b0 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:20.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc0a57f0 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9c2162700 10 monclient(hunting): authenticate will time out at 1361.553815s 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9bb7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fe9bc0ae8f0 0x7fe9bc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50738/0 (socket says 172.21.15.115:50738) 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9bb7fe700 10 monclient(hunting): get_auth_request con 0x7fe9bc0ae8f0 auth_method 0 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 1 -- 172.21.15.115:0/780038539 learned_addr learned my addr 172.21.15.115:0/780038539 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 10 monclient(hunting): get_auth_request con 0x7fe9bc0aecc0 auth_method 0 2023-09-30T01:08:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.103+0000 7fe9baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:20.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_done global_id 4953 payload 274 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 -- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 msgr2=0x7fe9bc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 -- 172.21.15.115:0/780038539 >> v1:172.21.15.157:6789/0 conn(0x7fe9bc1415c0 legacy=0x7fe9bc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 monclient: found mon.noname-a 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 -- 172.21.15.115:0/780038539 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0a5480 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9baffd700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9bb7fe700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a5d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098e0 tx=0x7fe9ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73158ba4cab18afd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 1 -- 172.21.15.115:0/780038539 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00b640 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: dump: 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:20.117 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 1 -- 172.21.15.115:0/780038539 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac00b7a0 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9c2162700 5 monclient: authenticate success, global_id 4953 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9c2162700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:20.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 1 -- 172.21.15.115:0/780038539 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00bb40 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9c2162700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9c2162700 4 set_mon_vals no callback set 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.104+0000 7fe9ba7fc700 10 monclient: dump: 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.119 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 10 monclient: shutdown 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 -- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0ae8f0 msgr2=0x7fe9bc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a5d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0098e0 tx=0x7fe9ac0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 -- 172.21.15.115:0/780038539 shutdown_connections 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a5d10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 --2- 172.21.15.115:0/780038539 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc0aecc0 0x7fe9bc0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 -- 172.21.15.115:0/780038539 >> 172.21.15.115:0/780038539 conn(0x7fe9bc06a460 msgr2=0x7fe9bc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 -- 172.21.15.115:0/780038539 shutdown_connections 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 -- 172.21.15.115:0/780038539 wait complete. 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 10 monclient: build_initial_monmap 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 10 monclient: monmap: 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.121 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 librados: starting msgr at 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.105+0000 7fe9c2162700 1 librados: starting objecter 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 Processor -- start 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 -- start start 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 librados: setting wanted keys 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 librados: calling monclient init 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient: init 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:20.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 0x7fe9bc14ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): picked mon.noname-c con 0x7fe9bc1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc14e290 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc14f1a0 con 0x7fe9bc0ae8f0 2023-09-30T01:08:20.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc0ac070 con 0x7fe9bc1415c0 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9c2162700 10 monclient(hunting): authenticate will time out at 1361.556822s 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 0x7fe9bc14ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40120/0 (socket says 172.21.15.115:40120) 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 1 -- 172.21.15.115:0/111682165 learned_addr learned my addr 172.21.15.115:0/111682165 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 0x7fe9bc14ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:50742/0 (socket says 172.21.15.115:50742) 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 10 monclient(hunting): get_auth_request con 0x7fe9bc0aecc0 auth_method 0 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:20.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 10 monclient(hunting): get_auth_request con 0x7fe9bc1415c0 auth_method 0 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.106+0000 7fe9bb7fe700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): get_auth_request con 0x7fe9bc0ae8f0 auth_method 0 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:20.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient(hunting): handle_auth_done global_id 4956 payload 824 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 -- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 msgr2=0x7fe9bc14ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 0x7fe9bc14ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 -- 172.21.15.115:0/111682165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9bc0ae8f0 msgr2=0x7fe9bc0a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient: _finish_hunting 0 2023-09-30T01:08:20.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 monclient: found mon.noname-a 2023-09-30T01:08:20.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:20.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 -- 172.21.15.115:0/111682165 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40094f0 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9baffd700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400ec50 tx=0x7fe9a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 client.?.objecter ms_handle_connect 0x7fe9bc0aecc0 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400c850 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: dump: 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:20.128 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:20.130 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 5 monclient: authenticate success, global_id 4956 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a400c9b0 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 10 monclient: _renew_subs 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac009420 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400cd50 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b37fe700 10 monclient: dump: 2023-09-30T01:08:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:20.133 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9b9ffb700 4 set_mon_vals no callback set 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 10 client.4956.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:20.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 10 monclient: _renew_subs 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc0aac30 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.107+0000 7fe9c2162700 1 librados: init done 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.108+0000 7fe9c2162700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.108+0000 7fe9c2162700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.108+0000 7fe9c2162700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.108+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc14e410 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.108+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe9a4013680 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 1 --2- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 0x7fe99c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9bb7fe700 1 --2- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 0x7fe99c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe9a40841c0 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9bb7fe700 10 monclient: get_auth_request con 0x7fe99c068890 auth_method 0 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 10 client.4956.objecter ms_dispatch 0x7fe9bc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:20.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 3 client.4956.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:20.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 3 client.4956.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:20.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 20 client.4956.objecter dump_active .. 0 homeless 2023-09-30T01:08:20.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9b37fe700 10 client.4956.objecter ms_handle_connect 0x7fe99c068890 2023-09-30T01:08:20.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.109+0000 7fe9bb7fe700 1 --2- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 0x7fe99c06ad40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0093f0 tx=0x7fe9ac017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:20.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.113+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9bc14e410 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.113+0000 7fe9b37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:20.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.113+0000 7fe9b37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.334+0000 7fe9c2162700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.334+0000 7fe9c2162700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.334+0000 7fe9c2162700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.334+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9bc14e410 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.335+0000 7fe9b37fe700 1 -- 172.21.15.115:0/111682165 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe9bc14e410 con 0x7fe9bc0aecc0 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.335+0000 7fe9b37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:20.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.335+0000 7fe9b37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:20.336 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:20.336 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 10 librados: watch_flush enter 2023-09-30T01:08:20.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 10 librados: watch_flush exit 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 20 client.4956.objecter shutdown clearing up homeless session... 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 10 client.4956.objecter successfully canceled tick 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 msgr2=0x7fe99c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 0x7fe99c06ad40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0093f0 tx=0x7fe9ac017000 comp rx=0 tx=0).stop 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 10 monclient: shutdown 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 msgr2=0x7fe9bc14e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400ec50 tx=0x7fe9a4009250 comp rx=0 tx=0).stop 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 shutdown_connections 2023-09-30T01:08:20.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> v2:172.21.15.115:6816/115635 conn(0x7fe99c068890 0x7fe99c06ad40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe9bc1415c0 0x7fe9bc14ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9bc0ae8f0 0x7fe9bc0a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 --2- 172.21.15.115:0/111682165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9bc0aecc0 0x7fe9bc14e6f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 >> 172.21.15.115:0/111682165 conn(0x7fe9bc06a460 msgr2=0x7fe9bc13f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 shutdown_connections 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 -- 172.21.15.115:0/111682165 wait complete. 2023-09-30T01:08:20.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:20.337+0000 7fe9c2162700 1 librados: shutdown 2023-09-30T01:08:23.349 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.478+0000 7fe0a0def700 10 monclient: get_monmap_and_config 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.478+0000 7fe0a0def700 10 monclient: build_initial_monmap 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.478+0000 7fe0a0def700 10 monclient: monmap: 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:23.479764+0000 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:23.479764+0000 2023-09-30T01:08:23.479 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.479+0000 7fe0a0def700 1 Processor -- start 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.479+0000 7fe0a0def700 1 -- start start 2023-09-30T01:08:23.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.479+0000 7fe0a0def700 10 monclient: init 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe09c0ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-b con 0x7fe09c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe09c1415c0 0x7fe09c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-c con 0x7fe09c1415c0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:23.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c0a7040 con 0x7fe09c0ae8f0 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09c0a57f0 con 0x7fe09c1415c0 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0a5480 con 0x7fe09c0aecc0 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient(hunting): authenticate will time out at 1364.930240s 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38224/0 (socket says 172.21.15.115:38224) 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 -- 172.21.15.115:0/1315111688 learned_addr learned my addr 172.21.15.115:0/1315111688 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): get_auth_request con 0x7fe09c0aecc0 auth_method 0 2023-09-30T01:08:23.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09ad9d700 1 --2- 172.21.15.115:0/1315111688 >> v2:172.21.15.157:3300/0 conn(0x7fe09c1415c0 0x7fe09c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09ad9d700 10 monclient(hunting): get_auth_request con 0x7fe09c1415c0 auth_method 0 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient(hunting): handle_auth_done global_id 4697 payload 274 2023-09-30T01:08:23.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 -- 172.21.15.115:0/1315111688 >> v2:172.21.15.157:3300/0 conn(0x7fe09c1415c0 msgr2=0x7fe09c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 --2- 172.21.15.115:0/1315111688 >> v2:172.21.15.157:3300/0 conn(0x7fe09c1415c0 0x7fe09c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 -- 172.21.15.115:0/1315111688 >> v1:172.21.15.157:6789/0 conn(0x7fe09c0ae8f0 legacy=0x7fe09c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 monclient: found mon.noname-b 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 -- 172.21.15.115:0/1315111688 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c0a55c0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe099d9b700 1 --2- 172.21.15.115:0/1315111688 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0099b0 tx=0x7fe08c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7035466db3e052 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 1 -- 172.21.15.115:0/1315111688 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00b820 con 0x7fe09c0aecc0 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:23.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: dump: 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: _finish_auth 0 2023-09-30T01:08:23.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 5 monclient: authenticate success, global_id 4697 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 1 -- 172.21.15.115:0/1315111688 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c00b980 con 0x7fe09c0aecc0 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 1 -- 172.21.15.115:0/1315111688 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00bd20 con 0x7fe09c0aecc0 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe0a0def700 4 set_mon_vals no callback set 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.480+0000 7fe09959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe09959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe09959a700 10 monclient: dump: 2023-09-30T01:08:23.494 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 10 monclient: shutdown 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:23.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 -- 172.21.15.115:0/1315111688 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 msgr2=0x7fe09c0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1315111688 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0099b0 tx=0x7fe08c009250 comp rx=0 tx=0).stop 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 -- 172.21.15.115:0/1315111688 shutdown_connections 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1315111688 >> v2:172.21.15.157:3300/0 conn(0x7fe09c1415c0 0x7fe09c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1315111688 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c0aecc0 0x7fe09c0a5d10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 -- 172.21.15.115:0/1315111688 >> 172.21.15.115:0/1315111688 conn(0x7fe09c06a460 msgr2=0x7fe09c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 -- 172.21.15.115:0/1315111688 shutdown_connections 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.481+0000 7fe0a0def700 1 -- 172.21.15.115:0/1315111688 wait complete. 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 10 monclient: build_initial_monmap 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 10 monclient: monmap: 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:23.496 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 librados: starting msgr at 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 librados: starting objecter 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 Processor -- start 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 -- start start 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 librados: setting wanted keys 2023-09-30T01:08:23.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 1 librados: calling monclient init 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.482+0000 7fe0a0def700 10 monclient: init 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe09c0ae8f0 0x7fe09c0a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-b con 0x7fe09c0ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-a con 0x7fe09c0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 0x7fe09c1513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): picked mon.noname-c con 0x7fe09c1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:23.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0a9a30 con 0x7fe09c0aecc0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0a96f0 con 0x7fe09c0ae8f0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0a8d00 con 0x7fe09c1415c0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient(hunting): authenticate will time out at 1364.933117s 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 0x7fe09c1513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57422/0 (socket says 172.21.15.115:57422) 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 0x7fe09c1513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38238/0 (socket says 172.21.15.115:38238) 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 -- 172.21.15.115:0/1002612979 learned_addr learned my addr 172.21.15.115:0/1002612979 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): get_auth_request con 0x7fe09c0aecc0 auth_method 0 2023-09-30T01:08:23.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): get_auth_request con 0x7fe09c1415c0 auth_method 0 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe09c0ae8f0 0x7fe09c0a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): get_auth_request con 0x7fe09c0ae8f0 auth_method 0 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:23.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe09a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient(hunting): handle_auth_done global_id 4959 payload 824 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 -- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 msgr2=0x7fe09c1513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 0x7fe09c1513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 -- 172.21.15.115:0/1002612979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe09c0ae8f0 msgr2=0x7fe09c0a91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe09c0ae8f0 0x7fe09c0a91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 monclient: found mon.noname-a 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 -- 172.21.15.115:0/1002612979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c0094f0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe099d9b700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009290 tx=0x7fe08c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 client.?.objecter ms_handle_connect 0x7fe09c0aecc0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c011c30 con 0x7fe09c0aecc0 2023-09-30T01:08:23.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: dump: 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 5 monclient: authenticate success, global_id 4959 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c020070 con 0x7fe09c0aecc0 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c01b450 con 0x7fe09c0aecc0 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _renew_subs 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c0a8f20 con 0x7fe09c0aecc0 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe098d99700 4 set_mon_vals no callback set 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0927fc700 10 monclient: dump: 2023-09-30T01:08:23.504 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 client.4959.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _renew_subs 2023-09-30T01:08:23.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c151ad0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.483+0000 7fe0a0def700 1 librados: init done 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.484+0000 7fe0a0def700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.484+0000 7fe0a0def700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.484+0000 7fe0a0def700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.484+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c0a9be0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe08c022030 con 0x7fe09c0aecc0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 1 --2- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 0x7fe07c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe08c085d20 con 0x7fe09c0aecc0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe09a59c700 1 --2- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 0x7fe07c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 10 client.4959.objecter ms_dispatch 0x7fe09c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 3 client.4959.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:23.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 3 client.4959.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe09a59c700 10 monclient: get_auth_request con 0x7fe07c068890 auth_method 0 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.485+0000 7fe0927fc700 20 client.4959.objecter dump_active .. 0 homeless 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.486+0000 7fe0927fc700 10 client.4959.objecter ms_handle_connect 0x7fe07c068890 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.486+0000 7fe09a59c700 1 --2- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 0x7fe07c06ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe084003a30 tx=0x7fe084007ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.490+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe09c0a9be0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.490+0000 7fe0927fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:23.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.490+0000 7fe0927fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:23.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0a0def700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0a0def700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0a0def700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe09c0a9be0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0927fc700 1 -- 172.21.15.115:0/1002612979 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe09c0a9be0 con 0x7fe09c0aecc0 2023-09-30T01:08:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0927fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:23.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.712+0000 7fe0927fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:23.715 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:23.715 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 10 librados: watch_flush enter 2023-09-30T01:08:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 10 librados: watch_flush exit 2023-09-30T01:08:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 20 client.4959.objecter shutdown clearing up homeless session... 2023-09-30T01:08:23.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 10 client.4959.objecter successfully canceled tick 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 msgr2=0x7fe07c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 0x7fe07c06ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe084003a30 tx=0x7fe084007ea0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 10 monclient: shutdown 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 msgr2=0x7fe09c0a87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009290 tx=0x7fe08c016000 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 shutdown_connections 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> v2:172.21.15.115:6816/115635 conn(0x7fe07c068890 0x7fe07c06ad40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fe09c1415c0 0x7fe09c1513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe09c0ae8f0 0x7fe09c0a91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 --2- 172.21.15.115:0/1002612979 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe09c0aecc0 0x7fe09c0a87c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 >> 172.21.15.115:0/1002612979 conn(0x7fe09c06a460 msgr2=0x7fe09c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 shutdown_connections 2023-09-30T01:08:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 -- 172.21.15.115:0/1002612979 wait complete. 2023-09-30T01:08:23.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:23.714+0000 7fe0a0def700 1 librados: shutdown 2023-09-30T01:08:26.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:26.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.854+0000 7f04d3164700 10 monclient: get_monmap_and_config 2023-09-30T01:08:26.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.854+0000 7f04d3164700 10 monclient: build_initial_monmap 2023-09-30T01:08:26.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.854+0000 7f04d3164700 10 monclient: monmap: 2023-09-30T01:08:26.855 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:26.855 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:26.856151+0000 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:26.856151+0000 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:26.856 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.854+0000 7f04d3164700 1 Processor -- start 2023-09-30T01:08:26.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 1 -- start start 2023-09-30T01:08:26.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient: init 2023-09-30T01:08:26.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:26.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-a con 0x7f04cc0ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04cc0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04cc1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0a7040 con 0x7f04cc1415c0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0a57f0 con 0x7f04cc0aecc0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc0a5480 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d3164700 10 monclient(hunting): authenticate will time out at 1368.306714s 2023-09-30T01:08:26.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57424/0 (socket says 172.21.15.115:57424) 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 1 -- 172.21.15.115:0/2622585747 learned_addr learned my addr 172.21.15.115:0/2622585747 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d1701700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f04cc1415c0 0x7f04cc0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:55064/0 (socket says 172.21.15.115:55064) 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): get_auth_request con 0x7f04cc0ae8f0 auth_method 0 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.855+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:26.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 10 monclient(hunting): handle_auth_done global_id 4962 payload 274 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 1 -- 172.21.15.115:0/2622585747 >> v1:172.21.15.157:6789/0 conn(0x7f04cc0aecc0 legacy=0x7f04cc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 1 -- 172.21.15.115:0/2622585747 >> v1:172.21.15.115:6789/0 conn(0x7f04cc1415c0 legacy=0x7f04cc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 10 monclient: _finish_hunting 0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 1 monclient: found mon.noname-a 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 1 -- 172.21.15.115:0/2622585747 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0a55c0 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 1 -- 172.21.15.115:0/2622585747 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53807609 0 0) 0x7f04cc0a7040 con 0x7f04cc1415c0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d0f00700 1 --2- 172.21.15.115:0/2622585747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f04c00099b0 tx=0x7f04c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79177041ff39fd88 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 1 -- 172.21.15.115:0/2622585747 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000b820 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: dump: 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 5 monclient: authenticate success, global_id 4962 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 1 -- 172.21.15.115:0/2622585747 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c000b980 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 4 set_mon_vals no callback set 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 1 -- 172.21.15.115:0/2622585747 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000bd20 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04cb7fe700 10 monclient: dump: 2023-09-30T01:08:26.869 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 10 monclient: shutdown 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:26.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 -- 172.21.15.115:0/2622585747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0ae8f0 msgr2=0x7f04cc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 --2- 172.21.15.115:0/2622585747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f04c00099b0 tx=0x7f04c0009250 comp rx=0 tx=0).stop 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 -- 172.21.15.115:0/2622585747 shutdown_connections 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 --2- 172.21.15.115:0/2622585747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0ae8f0 0x7f04cc0a5d10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 -- 172.21.15.115:0/2622585747 >> 172.21.15.115:0/2622585747 conn(0x7f04cc06a460 msgr2=0x7f04cc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 -- 172.21.15.115:0/2622585747 shutdown_connections 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.856+0000 7f04d3164700 1 -- 172.21.15.115:0/2622585747 wait complete. 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.857+0000 7f04d3164700 10 monclient: build_initial_monmap 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.857+0000 7f04d3164700 10 monclient: monmap: 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:26.871 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.857+0000 7f04d3164700 1 librados: starting msgr at 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.857+0000 7f04d3164700 1 librados: starting objecter 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.857+0000 7f04d3164700 1 Processor -- start 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 -- start start 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 librados: setting wanted keys 2023-09-30T01:08:26.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 librados: calling monclient init 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient: init 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-c con 0x7f04cc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-a con 0x7f04cc0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 0x7f04cc0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): picked mon.noname-b con 0x7f04cc1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:26.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0a8e90 con 0x7f04cc0aecc0 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0a7c10 con 0x7f04cc1415c0 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc152180 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d3164700 10 monclient(hunting): authenticate will time out at 1368.309604s 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:38242/0 (socket says 172.21.15.115:38242) 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 1 -- 172.21.15.115:0/3843960683 learned_addr learned my addr 172.21.15.115:0/3843960683 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57436/0 (socket says 172.21.15.115:57436) 2023-09-30T01:08:26.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): get_auth_request con 0x7f04cc0ae8f0 auth_method 0 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): get_auth_request con 0x7f04cc0aecc0 auth_method 0 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:26.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d1701700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 0x7f04cc0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.858+0000 7f04d0f00700 1 -- 172.21.15.115:0/3843960683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 msgr2=0x7f04cc0a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 0x7f04cc0a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 -- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 msgr2=0x7f04cc0a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 10 monclient: _finish_hunting 0 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 monclient: found mon.noname-c 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 -- 172.21.15.115:0/3843960683 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00094f0 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d1701700 10 monclient: get_auth_request con 0x7f04cc1415c0 auth_method 0 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d1701700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 0x7f04cc0a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04cbfff700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d0f00700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009290 tx=0x7f04c0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 client.?.objecter ms_handle_connect 0x7f04cc0ae8f0 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000bbc0 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: dump: 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c0020070 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 5 monclient: authenticate success, global_id 4703 2023-09-30T01:08:26.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 10 monclient: _renew_subs 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc13d790 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c001b6a0 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04c8ff9700 10 monclient: dump: 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:26.879 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04caffd700 4 set_mon_vals no callback set 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 10 monclient: _renew_subs 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc0a8a10 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.859+0000 7f04d3164700 1 librados: init done 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.860+0000 7f04d3164700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:26.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.860+0000 7f04d3164700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.860+0000 7f04d3164700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.860+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc072960 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.860+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f04c0022030 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 1 --2- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 0x7f04b006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04cbfff700 1 --2- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 0x7f04b006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04cbfff700 10 monclient: get_auth_request con 0x7f04b0068890 auth_method 0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f04c0085d80 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 10 client.4703.objecter ms_dispatch 0x7f04cc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 3 client.4703.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 3 client.4703.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:26.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 20 client.4703.objecter dump_active .. 0 homeless 2023-09-30T01:08:26.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04c8ff9700 10 client.4703.objecter ms_handle_connect 0x7f04b0068890 2023-09-30T01:08:26.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.861+0000 7f04cbfff700 1 --2- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 0x7f04b006ad40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0aab00 tx=0x7f04b8007ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:26.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.865+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04c0053640 con 0x7f04cc0ae8f0 2023-09-30T01:08:26.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.865+0000 7f04c8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:26.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:26.865+0000 7f04c8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04d3164700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04d3164700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04d3164700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04cc072960 con 0x7f04cc0ae8f0 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04c8ff9700 1 -- 172.21.15.115:0/3843960683 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f04c0056c60 con 0x7f04cc0ae8f0 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04c8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.091+0000 7f04c8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:27.093 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:27.094 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:27.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.093+0000 7f04d3164700 10 librados: watch_flush enter 2023-09-30T01:08:27.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.093+0000 7f04d3164700 10 librados: watch_flush exit 2023-09-30T01:08:27.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.093+0000 7f04d3164700 20 client.4703.objecter shutdown clearing up homeless session... 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.093+0000 7f04d3164700 10 client.4703.objecter successfully canceled tick 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 msgr2=0x7f04b006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 0x7f04b006ad40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0aab00 tx=0x7f04b8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 10 monclient: shutdown 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 msgr2=0x7f04cc151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009290 tx=0x7f04c0016000 comp rx=0 tx=0).stop 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 shutdown_connections 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> v2:172.21.15.115:6816/115635 conn(0x7f04b0068890 0x7f04b006ad40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04cc1415c0 0x7f04cc0a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f04cc0ae8f0 0x7f04cc151650 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:27.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 --2- 172.21.15.115:0/3843960683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04cc0aecc0 0x7f04cc0a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:27.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 >> 172.21.15.115:0/3843960683 conn(0x7f04cc06a460 msgr2=0x7f04cc13fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:27.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 shutdown_connections 2023-09-30T01:08:27.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 -- 172.21.15.115:0/3843960683 wait complete. 2023-09-30T01:08:27.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:27.094+0000 7f04d3164700 1 librados: shutdown 2023-09-30T01:08:30.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.235+0000 7f2c88308700 10 monclient: get_monmap_and_config 2023-09-30T01:08:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.235+0000 7f2c88308700 10 monclient: build_initial_monmap 2023-09-30T01:08:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.235+0000 7f2c88308700 10 monclient: monmap: 2023-09-30T01:08:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:30.236336+0000 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:30.236336+0000 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:30.236 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.235+0000 7f2c88308700 1 Processor -- start 2023-09-30T01:08:30.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 -- start start 2023-09-30T01:08:30.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient: init 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c800ae8f0 0x7f2c800a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-c con 0x7f2c800ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2c800aecc0 0x7f2c800b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-a con 0x7f2c800aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c801415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c800a7040 con 0x7f2c801415c0 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:30.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c800b19b0 con 0x7f2c800aecc0 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c800a57f0 con 0x7f2c800ae8f0 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c88308700 10 monclient(hunting): authenticate will time out at 1371.686771s 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2c800aecc0 0x7f2c800b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2c800aecc0 0x7f2c800b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57450/0 (socket says 172.21.15.115:57450) 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 1 -- 172.21.15.115:0/1731307642 learned_addr learned my addr 172.21.15.115:0/1731307642 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 10 monclient(hunting): get_auth_request con 0x7f2c800aecc0 auth_method 0 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:30.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.236+0000 7f2c858a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c860a4700 1 --2- 172.21.15.115:0/1731307642 >> v2:172.21.15.157:3300/0 conn(0x7f2c800ae8f0 0x7f2c800a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c860a4700 10 monclient(hunting): get_auth_request con 0x7f2c800ae8f0 auth_method 0 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c860a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c860a4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c860a4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient(hunting): handle_auth_done global_id 4971 payload 274 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 -- 172.21.15.115:0/1731307642 >> v2:172.21.15.157:3300/0 conn(0x7f2c800ae8f0 msgr2=0x7f2c800a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 --2- 172.21.15.115:0/1731307642 >> v2:172.21.15.157:3300/0 conn(0x7f2c800ae8f0 0x7f2c800a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 -- 172.21.15.115:0/1731307642 >> v1:172.21.15.157:6789/0 conn(0x7f2c801415c0 legacy=0x7f2c800adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient: _finish_hunting 0 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 monclient: found mon.noname-a 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 -- 172.21.15.115:0/1731307642 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c800a5480 con 0x7f2c800aecc0 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c858a3700 1 --2- 172.21.15.115:0/1731307642 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c800b1470 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2c800b59c0 tx=0x7f2c7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be7cae52dd6824e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 1 -- 172.21.15.115:0/1731307642 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00b860 con 0x7f2c800aecc0 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: dump: 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:30.249 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: _finish_auth 0 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:30.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 1 -- 172.21.15.115:0/1731307642 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c00b9c0 con 0x7f2c800aecc0 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c88308700 5 monclient: authenticate success, global_id 4971 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c88308700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c88308700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 1 -- 172.21.15.115:0/1731307642 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c00bd60 con 0x7f2c800aecc0 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c88308700 4 set_mon_vals no callback set 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.237+0000 7f2c850a2700 10 monclient: dump: 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 10 monclient: shutdown 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 -- 172.21.15.115:0/1731307642 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 msgr2=0x7f2c800b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:30.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 --2- 172.21.15.115:0/1731307642 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c800b1470 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2c800b59c0 tx=0x7f2c7c009250 comp rx=0 tx=0).stop 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 -- 172.21.15.115:0/1731307642 shutdown_connections 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 --2- 172.21.15.115:0/1731307642 >> v2:172.21.15.157:3300/0 conn(0x7f2c800ae8f0 0x7f2c800a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 --2- 172.21.15.115:0/1731307642 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c800b1470 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 -- 172.21.15.115:0/1731307642 >> 172.21.15.115:0/1731307642 conn(0x7f2c8006a460 msgr2=0x7f2c800630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 -- 172.21.15.115:0/1731307642 shutdown_connections 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 -- 172.21.15.115:0/1731307642 wait complete. 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 10 monclient: build_initial_monmap 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 10 monclient: monmap: 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 librados: starting msgr at 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.238+0000 7f2c88308700 1 librados: starting objecter 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 Processor -- start 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 -- start start 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 librados: setting wanted keys 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 librados: calling monclient init 2023-09-30T01:08:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient: init 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2c800ae8f0 0x7f2c8013e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-c con 0x7f2c800ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-a con 0x7f2c800aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c801415c0 0x7f2c800a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): picked mon.noname-b con 0x7f2c801415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80151e10 con 0x7f2c800aecc0 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800ab260 con 0x7f2c801415c0 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80151920 con 0x7f2c800ae8f0 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c88308700 10 monclient(hunting): authenticate will time out at 1371.689714s 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2c800ae8f0 0x7f2c8013e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57466/0 (socket says 172.21.15.115:57466) 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 1 -- 172.21.15.115:0/2305664567 learned_addr learned my addr 172.21.15.115:0/2305664567 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): get_auth_request con 0x7f2c800aecc0 auth_method 0 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): get_auth_request con 0x7f2c800ae8f0 auth_method 0 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c858a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c868a5700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c801415c0 0x7f2c800a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c868a5700 10 monclient(hunting): get_auth_request con 0x7f2c801415c0 auth_method 0 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c868a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c868a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c868a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.239+0000 7f2c860a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c868a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c868a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c868a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 10 monclient(hunting): handle_auth_done global_id 4974 payload 824 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 -- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2c800ae8f0 msgr2=0x7f2c8013e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2c800ae8f0 0x7f2c8013e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 -- 172.21.15.115:0/2305664567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c801415c0 msgr2=0x7f2c800a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c801415c0 0x7f2c800a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 10 monclient: _finish_hunting 0 2023-09-30T01:08:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 monclient: found mon.noname-a 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 -- 172.21.15.115:0/2305664567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c0094f0 con 0x7f2c800aecc0 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c858a3700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c800b0920 tx=0x7f2c7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 client.?.objecter ms_handle_connect 0x7f2c800aecc0 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c024440 con 0x7f2c800aecc0 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: dump: 2023-09-30T01:08:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c0245a0 con 0x7f2c800aecc0 2023-09-30T01:08:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 5 monclient: authenticate success, global_id 4974 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 10 monclient: _renew_subs 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c80151b40 con 0x7f2c800aecc0 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c024960 con 0x7f2c800aecc0 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c767fc700 10 monclient: dump: 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c848a1700 4 set_mon_vals no callback set 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 10 client.4974.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 10 monclient: _renew_subs 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c800a9a40 con 0x7f2c800aecc0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.240+0000 7f2c88308700 1 librados: init done 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.241+0000 7f2c88308700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.241+0000 7f2c88308700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.241+0000 7f2c88308700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.241+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c80072960 con 0x7f2c800aecc0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.241+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2c7c01a420 con 0x7f2c800aecc0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 1 --2- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 0x7f2c6406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2c7c085d10 con 0x7f2c800aecc0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 10 client.4974.objecter ms_dispatch 0x7f2c80001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 3 client.4974.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 3 client.4974.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:30.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c860a4700 1 --2- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 0x7f2c6406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c860a4700 10 monclient: get_auth_request con 0x7f2c64068890 auth_method 0 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 20 client.4974.objecter dump_active .. 0 homeless 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c860a4700 1 --2- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 0x7f2c6406ad40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006fd0 tx=0x7f2c70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.242+0000 7f2c767fc700 10 client.4974.objecter ms_handle_connect 0x7f2c64068890 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.246+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c7c0535d0 con 0x7f2c800aecc0 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.246+0000 7f2c767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.246+0000 7f2c767fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c88308700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c88308700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c88308700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:30.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2c8006f840 con 0x7f2c800aecc0 2023-09-30T01:08:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c767fc700 1 -- 172.21.15.115:0/2305664567 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2c7c056bf0 con 0x7f2c800aecc0 2023-09-30T01:08:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.467+0000 7f2c767fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:30.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:30.468 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:30.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 10 librados: watch_flush enter 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 10 librados: watch_flush exit 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 20 client.4974.objecter shutdown clearing up homeless session... 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 10 client.4974.objecter successfully canceled tick 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 msgr2=0x7f2c6406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 0x7f2c6406ad40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006fd0 tx=0x7f2c70008040 comp rx=0 tx=0).stop 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 10 monclient: shutdown 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 msgr2=0x7f2c801513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2c800b0920 tx=0x7f2c7c017000 comp rx=0 tx=0).stop 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 shutdown_connections 2023-09-30T01:08:30.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> v2:172.21.15.115:6816/115635 conn(0x7f2c64068890 0x7f2c6406ad40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c801415c0 0x7f2c800a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2c800ae8f0 0x7f2c8013e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 --2- 172.21.15.115:0/2305664567 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c800aecc0 0x7f2c801513e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 >> 172.21.15.115:0/2305664567 conn(0x7f2c8006a460 msgr2=0x7f2c800630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 shutdown_connections 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 -- 172.21.15.115:0/2305664567 wait complete. 2023-09-30T01:08:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:30.469+0000 7f2c88308700 1 librados: shutdown 2023-09-30T01:08:33.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:33.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.610+0000 7fd1ea4af700 10 monclient: get_monmap_and_config 2023-09-30T01:08:33.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.610+0000 7fd1ea4af700 10 monclient: build_initial_monmap 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.610+0000 7fd1ea4af700 10 monclient: monmap: 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:33.611626+0000 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:33.611626+0000 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:33.611 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:33.612 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.611+0000 7fd1ea4af700 1 Processor -- start 2023-09-30T01:08:33.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.611+0000 7fd1ea4af700 1 -- start start 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.611+0000 7fd1ea4af700 10 monclient: init 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 0x7fd1e40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1e40aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1e41415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e40a7040 con 0x7fd1e40aecc0 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e40a57f0 con 0x7fd1e41415c0 2023-09-30T01:08:33.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e40a5480 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient(hunting): authenticate will time out at 1375.062159s 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e37fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd1e40aecc0 0x7fd1e40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:43254/0 (socket says 172.21.15.115:43254) 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e37fe700 1 -- 172.21.15.115:0/1287122339 learned_addr learned my addr 172.21.15.115:0/1287122339 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 0x7fd1e40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): get_auth_request con 0x7fd1e40ae8f0 auth_method 0 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:33.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 1 -- 172.21.15.115:0/1287122339 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085437257 0 0) 0x7fd1e40a7040 con 0x7fd1e40aecc0 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient(hunting): my global_id is 4977 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 1 -- 172.21.15.115:0/1287122339 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc003570 con 0x7fd1e40aecc0 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_done global_id 4709 payload 274 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 -- 172.21.15.115:0/1287122339 >> v1:172.21.15.157:6789/0 conn(0x7fd1e41415c0 legacy=0x7fd1e40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 -- 172.21.15.115:0/1287122339 >> v1:172.21.15.115:6789/0 conn(0x7fd1e40aecc0 legacy=0x7fd1e40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:08:33.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 monclient: found mon.noname-b 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 -- 172.21.15.115:0/1287122339 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e40a55c0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e3fff700 1 --2- 172.21.15.115:0/1287122339 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 0x7fd1e40a5d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009910 tx=0x7fd1d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac6d2907338f2f14 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 1 -- 172.21.15.115:0/1287122339 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800b640 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: dump: 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.624 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 1 -- 172.21.15.115:0/1287122339 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d800b7a0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 5 monclient: authenticate success, global_id 4709 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:33.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 1 -- 172.21.15.115:0/1287122339 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800bb60 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1ea4af700 4 set_mon_vals no callback set 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.612+0000 7fd1e2ffd700 10 monclient: dump: 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.627 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:33.627 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:33.627 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 10 monclient: shutdown 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 -- 172.21.15.115:0/1287122339 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 msgr2=0x7fd1e40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/1287122339 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 0x7fd1e40a5d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009910 tx=0x7fd1d80091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 -- 172.21.15.115:0/1287122339 shutdown_connections 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/1287122339 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40ae8f0 0x7fd1e40a5d10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 -- 172.21.15.115:0/1287122339 >> 172.21.15.115:0/1287122339 conn(0x7fd1e406a460 msgr2=0x7fd1e40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 -- 172.21.15.115:0/1287122339 shutdown_connections 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 1 -- 172.21.15.115:0/1287122339 wait complete. 2023-09-30T01:08:33.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 10 monclient: build_initial_monmap 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.613+0000 7fd1ea4af700 10 monclient: monmap: 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 librados: starting msgr at 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 librados: starting objecter 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 Processor -- start 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 -- start start 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 librados: setting wanted keys 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 1 librados: calling monclient init 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.614+0000 7fd1ea4af700 10 monclient: init 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-a con 0x7fd1e40ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 0x7fd1e40a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:33.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-c con 0x7fd1e40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e41415c0 0x7fd1e40abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e40aa6f0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e40a8590 con 0x7fd1e41415c0 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e413f070 con 0x7fd1e40aecc0 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1ea4af700 10 monclient(hunting): authenticate will time out at 1375.065212s 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:33.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40276/0 (socket says 172.21.15.115:40276) 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 -- 172.21.15.115:0/4003063894 learned_addr learned my addr 172.21.15.115:0/4003063894 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): get_auth_request con 0x7fd1e40ae8f0 auth_method 0 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 0x7fd1e40a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): get_auth_request con 0x7fd1e40aecc0 auth_method 0 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:33.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient(hunting): handle_auth_done global_id 4980 payload 824 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 -- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 msgr2=0x7fd1e40a7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 0x7fd1e40a7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 -- 172.21.15.115:0/4003063894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e41415c0 msgr2=0x7fd1e40abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e8a4c700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e41415c0 0x7fd1e40abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e41415c0 0x7fd1e40abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 monclient: found mon.noname-a 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e3fff700 1 -- 172.21.15.115:0/4003063894 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8009420 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.615+0000 7fd1e37fe700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 0x7fd1e40a7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1e3fff700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80091e0 tx=0x7fd1d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 client.?.objecter ms_handle_connect 0x7fd1e40ae8f0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800b910 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:33.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: dump: 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: _finish_auth 0 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 5 monclient: authenticate success, global_id 4980 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 10 monclient: _renew_subs 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d8020070 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e40a81a0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d801b450 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1cbfff700 10 monclient: dump: 2023-09-30T01:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:33.637 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1e27fc700 4 set_mon_vals no callback set 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 10 client.4980.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 10 monclient: _renew_subs 2023-09-30T01:08:33.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e40ac510 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.616+0000 7fd1ea4af700 1 librados: init done 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1ea4af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1ea4af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1ea4af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e40abbe0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd1d8022030 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.617+0000 7fd1cbfff700 1 --2- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 0x7fd1d006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd1d8085d30 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 10 client.4980.objecter ms_dispatch 0x7fd1e4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:33.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 3 client.4980.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 3 client.4980.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1e37fe700 1 --2- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 0x7fd1d006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1e37fe700 10 monclient: get_auth_request con 0x7fd1d0068890 auth_method 0 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 20 client.4980.objecter dump_active .. 0 homeless 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1cbfff700 10 client.4980.objecter ms_handle_connect 0x7fd1d0068890 2023-09-30T01:08:33.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.618+0000 7fd1e37fe700 1 --2- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 0x7fd1d006ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40a8c20 tx=0x7fd1d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:33.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.622+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1d80535f0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.622+0000 7fd1cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:33.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.622+0000 7fd1cbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.848+0000 7fd1ea4af700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.848+0000 7fd1ea4af700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.848+0000 7fd1ea4af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.848+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd1e40abbe0 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.849+0000 7fd1cbfff700 1 -- 172.21.15.115:0/4003063894 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd1d8056c10 con 0x7fd1e40ae8f0 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.849+0000 7fd1cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.849+0000 7fd1cbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:33.850 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:33.850 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 10 librados: watch_flush enter 2023-09-30T01:08:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 10 librados: watch_flush exit 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 20 client.4980.objecter shutdown clearing up homeless session... 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 10 client.4980.objecter successfully canceled tick 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 msgr2=0x7fd1d006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 0x7fd1d006ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40a8c20 tx=0x7fd1d4009040 comp rx=0 tx=0).stop 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 10 monclient: shutdown 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 msgr2=0x7fd1e413e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80091e0 tx=0x7fd1d8016000 comp rx=0 tx=0).stop 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 shutdown_connections 2023-09-30T01:08:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> v2:172.21.15.115:6816/115635 conn(0x7fd1d0068890 0x7fd1d006ad40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1e41415c0 0x7fd1e40abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd1e40ae8f0 0x7fd1e413e490 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 --2- 172.21.15.115:0/4003063894 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fd1e40aecc0 0x7fd1e40a7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 >> 172.21.15.115:0/4003063894 conn(0x7fd1e406a460 msgr2=0x7fd1e413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 shutdown_connections 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 -- 172.21.15.115:0/4003063894 wait complete. 2023-09-30T01:08:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:33.851+0000 7fd1ea4af700 1 librados: shutdown 2023-09-30T01:08:36.864 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.991+0000 7fcc43c64700 10 monclient: get_monmap_and_config 2023-09-30T01:08:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.991+0000 7fcc43c64700 10 monclient: build_initial_monmap 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.991+0000 7fcc43c64700 10 monclient: monmap: 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:36.992582+0000 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:36.992582+0000 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:36.992 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:36.993 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:36.993 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.992+0000 7fcc43c64700 1 Processor -- start 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.992+0000 7fcc43c64700 1 -- start start 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.992+0000 7fcc43c64700 10 monclient: init 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc3c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc3c0aecc0 0x7fcc3c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-c con 0x7fcc3c0aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcc3c1415c0 0x7fcc3c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-a con 0x7fcc3c1415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c0a7040 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c0a57f0 con 0x7fcc3c1415c0 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c0a5480 con 0x7fcc3c0aecc0 2023-09-30T01:08:37.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc43c64700 10 monclient(hunting): authenticate will time out at 1378.443520s 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc41a00700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fcc3c0ae8f0 0x7fcc3c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:43256/0 (socket says 172.21.15.115:43256) 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc41a00700 1 -- 172.21.15.115:0/3103200040 learned_addr learned my addr 172.21.15.115:0/3103200040 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 1 --2- 172.21.15.115:0/3103200040 >> v2:172.21.15.115:3300/0 conn(0x7fcc3c1415c0 0x7fcc3c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): get_auth_request con 0x7fcc3c1415c0 auth_method 0 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720140882 0 0) 0x7fcc3c0a7040 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc409fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc409fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc409fe700 10 monclient(hunting): my global_id is 4983 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28003570 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 1 --2- 172.21.15.115:0/3103200040 >> v2:172.21.15.157:3300/0 conn(0x7fcc3c0aecc0 0x7fcc3c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): get_auth_request con 0x7fcc3c0aecc0 auth_method 0 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc42201700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:37.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.993+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 10 monclient(hunting): handle_auth_done global_id 4986 payload 274 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 -- 172.21.15.115:0/3103200040 >> v2:172.21.15.157:3300/0 conn(0x7fcc3c0aecc0 msgr2=0x7fcc3c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 --2- 172.21.15.115:0/3103200040 >> v2:172.21.15.157:3300/0 conn(0x7fcc3c0aecc0 0x7fcc3c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 -- 172.21.15.115:0/3103200040 >> v1:172.21.15.115:6789/0 conn(0x7fcc3c0ae8f0 legacy=0x7fcc3c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860448427 0 0) 0x7fcc28003570 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 10 monclient: _finish_hunting 0 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 monclient: found mon.noname-a 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 -- 172.21.15.115:0/3103200040 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c0a55c0 con 0x7fcc3c1415c0 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc42201700 1 --2- 172.21.15.115:0/3103200040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c0adb60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c064080 tx=0x7fcc34009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73c703ca4eb15e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400e070 con 0x7fcc3c1415c0 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: dump: 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.007 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3400b7a0 con 0x7fcc3c1415c0 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc43c64700 5 monclient: authenticate success, global_id 4986 2023-09-30T01:08:37.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc43c64700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc43c64700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 1 -- 172.21.15.115:0/3103200040 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38001470 con 0x7fcc3c1415c0 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc43c64700 4 set_mon_vals no callback set 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.994+0000 7fcc409fe700 10 monclient: dump: 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 10 monclient: shutdown 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 -- 172.21.15.115:0/3103200040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c1415c0 msgr2=0x7fcc3c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 --2- 172.21.15.115:0/3103200040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c0adb60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c064080 tx=0x7fcc34009780 comp rx=0 tx=0).stop 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 -- 172.21.15.115:0/3103200040 shutdown_connections 2023-09-30T01:08:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 --2- 172.21.15.115:0/3103200040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c0adb60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 --2- 172.21.15.115:0/3103200040 >> v2:172.21.15.157:3300/0 conn(0x7fcc3c0aecc0 0x7fcc3c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 -- 172.21.15.115:0/3103200040 >> 172.21.15.115:0/3103200040 conn(0x7fcc3c06a460 msgr2=0x7fcc3c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 -- 172.21.15.115:0/3103200040 shutdown_connections 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 1 -- 172.21.15.115:0/3103200040 wait complete. 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 10 monclient: build_initial_monmap 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.995+0000 7fcc43c64700 10 monclient: monmap: 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:37.011 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 librados: starting msgr at 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 librados: starting objecter 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 Processor -- start 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 -- start start 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 librados: setting wanted keys 2023-09-30T01:08:37.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 1 librados: calling monclient init 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.996+0000 7fcc43c64700 10 monclient: init 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-a con 0x7fcc3c0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 0x7fcc3c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-c con 0x7fcc3c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:37.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): picked mon.noname-b con 0x7fcc3c1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:37.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:37.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c13eda0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:37.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0a93e0 con 0x7fcc3c1415c0 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c151a70 con 0x7fcc3c0aecc0 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc43c64700 10 monclient(hunting): authenticate will time out at 1378.447349s 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40296/0 (socket says 172.21.15.115:40296) 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 -- 172.21.15.115:0/117953067 learned_addr learned my addr 172.21.15.115:0/117953067 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): get_auth_request con 0x7fcc3c0ae8f0 auth_method 0 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 0x7fcc3c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): get_auth_request con 0x7fcc3c0aecc0 auth_method 0 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:37.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc42201700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc42201700 10 monclient(hunting): get_auth_request con 0x7fcc3c1415c0 auth_method 0 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc42201700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc42201700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc42201700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient(hunting): handle_auth_done global_id 4989 payload 824 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 -- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 msgr2=0x7fcc3c151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 0x7fcc3c151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 -- 172.21.15.115:0/117953067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc3c1415c0 msgr2=0x7fcc3c135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c135730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient: _finish_hunting 0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 monclient: found mon.noname-a 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 -- 172.21.15.115:0/117953067 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34009a20 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc411ff700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 0x7fcc3c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc41a00700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800cb90 tx=0x7fcc38001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc31ffb700 10 client.?.objecter ms_handle_connect 0x7fcc3c0ae8f0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.997+0000 7fcc31ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38007360 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:37.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: dump: 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:37.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 5 monclient: authenticate success, global_id 4989 2023-09-30T01:08:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc380074e0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 10 monclient: _renew_subs 2023-09-30T01:08:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c13e9a0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:37.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38004830 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:37.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc31ffb700 10 monclient: dump: 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:37.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc33fff700 4 set_mon_vals no callback set 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 10 client.4989.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 10 monclient: _renew_subs 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c0a77b0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.998+0000 7fcc43c64700 1 librados: init done 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc43c64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc43c64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc43c64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c072960 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcc38004ba0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:36.999+0000 7fcc31ffb700 1 --2- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 0x7fcc2006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:37.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcc38083760 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 10 client.4989.objecter ms_dispatch 0x7fcc3c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 3 client.4989.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 3 client.4989.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc411ff700 1 --2- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 0x7fcc2006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc411ff700 10 monclient: get_auth_request con 0x7fcc20068890 auth_method 0 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 20 client.4989.objecter dump_active .. 0 homeless 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc31ffb700 10 client.4989.objecter ms_handle_connect 0x7fcc20068890 2023-09-30T01:08:37.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.000+0000 7fcc411ff700 1 --2- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 0x7fcc2006ad40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0b4e10 tx=0x7fcc2c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:37.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.004+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc380510a0 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.004+0000 7fcc31ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:37.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.004+0000 7fcc31ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc43c64700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc43c64700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc43c64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:37.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc3c141a40 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc31ffb700 1 -- 172.21.15.115:0/117953067 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc38018100 con 0x7fcc3c0ae8f0 2023-09-30T01:08:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc31ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:37.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.226+0000 7fcc31ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:37.227 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:37.227 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:37.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 10 librados: watch_flush enter 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 10 librados: watch_flush exit 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 20 client.4989.objecter shutdown clearing up homeless session... 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 10 client.4989.objecter successfully canceled tick 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 msgr2=0x7fcc2006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 0x7fcc2006ad40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0b4e10 tx=0x7fcc2c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 10 monclient: shutdown 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 msgr2=0x7fcc3c150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800cb90 tx=0x7fcc38001b50 comp rx=0 tx=0).stop 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 shutdown_connections 2023-09-30T01:08:37.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> v2:172.21.15.115:6816/115635 conn(0x7fcc20068890 0x7fcc2006ad40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc3c1415c0 0x7fcc3c135730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc3c0ae8f0 0x7fcc3c150ea0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 --2- 172.21.15.115:0/117953067 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcc3c0aecc0 0x7fcc3c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 >> 172.21.15.115:0/117953067 conn(0x7fcc3c06a460 msgr2=0x7fcc3c138c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 shutdown_connections 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 -- 172.21.15.115:0/117953067 wait complete. 2023-09-30T01:08:37.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:37.228+0000 7fcc43c64700 1 librados: shutdown 2023-09-30T01:08:40.241 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:40.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 10 monclient: get_monmap_and_config 2023-09-30T01:08:40.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 10 monclient: build_initial_monmap 2023-09-30T01:08:40.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 10 monclient: monmap: 2023-09-30T01:08:40.369 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:40.370058+0000 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:40.370058+0000 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:40.370 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:40.371 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 1 Processor -- start 2023-09-30T01:08:40.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 1 -- start start 2023-09-30T01:08:40.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.369+0000 7f7905742700 10 monclient: init 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79000ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 0x7f79000a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): picked mon.noname-b con 0x7f79000aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79001415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79000a7040 con 0x7f79000ae8f0 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79000a57f0 con 0x7f79001415c0 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79000a5480 con 0x7f79000aecc0 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f7905742700 10 monclient(hunting): authenticate will time out at 1381.820350s 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78feffd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f79000ae8f0 0x7f79000b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:43272/0 (socket says 172.21.15.115:43272) 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 0x7f79000a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78feffd700 1 -- 172.21.15.115:0/1989336257 learned_addr learned my addr 172.21.15.115:0/1989336257 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:40.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): get_auth_request con 0x7f79000aecc0 auth_method 0 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fdffb700 1 -- 172.21.15.115:0/1989336257 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924656951 0 0) 0x7f79000a7040 con 0x7f79000ae8f0 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fdffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fdffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fdffb700 10 monclient(hunting): my global_id is 4992 2023-09-30T01:08:40.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fdffb700 1 -- 172.21.15.115:0/1989336257 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e8003570 con 0x7f79000ae8f0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_done global_id 4718 payload 274 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1989336257 >> v1:172.21.15.157:6789/0 conn(0x7f79001415c0 legacy=0x7f79000adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1989336257 >> v1:172.21.15.115:6789/0 conn(0x7f79000ae8f0 legacy=0x7f79000b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 monclient: found mon.noname-b 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1989336257 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000a55c0 con 0x7f79000aecc0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.370+0000 7f78fe7fc700 1 --2- 172.21.15.115:0/1989336257 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 0x7f79000a5d10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009910 tx=0x7f78f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70e080d782ff4f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 1 -- 172.21.15.115:0/1989336257 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000b640 con 0x7f79000aecc0 2023-09-30T01:08:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: dump: 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.383 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 1 -- 172.21.15.115:0/1989336257 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f000b7a0 con 0x7f79000aecc0 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 5 monclient: authenticate success, global_id 4718 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 4 set_mon_vals no callback set 2023-09-30T01:08:40.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 1 -- 172.21.15.115:0/1989336257 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000bb60 con 0x7f79000aecc0 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f78fdffb700 10 monclient: dump: 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.385 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 10 monclient: shutdown 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 -- 172.21.15.115:0/1989336257 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 msgr2=0x7f79000a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 --2- 172.21.15.115:0/1989336257 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 0x7f79000a5d10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009910 tx=0x7f78f00091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 -- 172.21.15.115:0/1989336257 shutdown_connections 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 --2- 172.21.15.115:0/1989336257 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000aecc0 0x7f79000a5d10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 -- 172.21.15.115:0/1989336257 >> 172.21.15.115:0/1989336257 conn(0x7f790006a460 msgr2=0x7f79000630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 -- 172.21.15.115:0/1989336257 shutdown_connections 2023-09-30T01:08:40.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.371+0000 7f7905742700 1 -- 172.21.15.115:0/1989336257 wait complete. 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 10 monclient: build_initial_monmap 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 10 monclient: monmap: 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:40.387 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 librados: starting msgr at 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 librados: starting objecter 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 Processor -- start 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 -- start start 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 librados: setting wanted keys 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 1 librados: calling monclient init 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.372+0000 7f7905742700 10 monclient: init 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:40.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 0x7f790013e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): picked mon.noname-c con 0x7f79000ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): picked mon.noname-a con 0x7f79000aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79001415c0 0x7f790013ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): picked mon.noname-b con 0x7f79001415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000aa240 con 0x7f79000aecc0 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000a9f00 con 0x7f79001415c0 2023-09-30T01:08:40.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79000a93a0 con 0x7f79000ae8f0 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f7905742700 10 monclient(hunting): authenticate will time out at 1381.823373s 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 0x7f790013e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40304/0 (socket says 172.21.15.115:40304) 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1913653807 learned_addr learned my addr 172.21.15.115:0/1913653807 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 0x7f790013e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:49420/0 (socket says 172.21.15.115:49420) 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): get_auth_request con 0x7f79000aecc0 auth_method 0 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): get_auth_request con 0x7f79000ae8f0 auth_method 0 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient(hunting): handle_auth_done global_id 4995 payload 824 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 msgr2=0x7f790013e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 0x7f790013e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1913653807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79001415c0 msgr2=0x7f790013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78ff7fe700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79001415c0 0x7f790013ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79001415c0 0x7f790013ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 monclient: found mon.noname-a 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 -- 172.21.15.115:0/1913653807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0009420 con 0x7f79000aecc0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78fe7fc700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f78f00091e0 tx=0x7f78f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78e6ffd700 10 client.?.objecter ms_handle_connect 0x7f79000aecc0 2023-09-30T01:08:40.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.373+0000 7f78e6ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000b910 con 0x7f79000aecc0 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: dump: 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.393 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0020070 con 0x7f79000aecc0 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 5 monclient: authenticate success, global_id 4995 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 10 monclient: _renew_subs 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79000a9560 con 0x7f79000aecc0 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f001b450 con 0x7f79000aecc0 2023-09-30T01:08:40.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78e6ffd700 10 monclient: dump: 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.395 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 10 client.4995.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 10 monclient: _renew_subs 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f78fd7fa700 4 set_mon_vals no callback set 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790013b2f0 con 0x7f79000aecc0 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.374+0000 7f7905742700 1 librados: init done 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f7905742700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f7905742700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f7905742700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79000aa3b0 con 0x7f79000aecc0 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f78f0022030 con 0x7f79000aecc0 2023-09-30T01:08:40.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.375+0000 7f78e6ffd700 1 --2- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 0x7f78ec06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f78f0085d70 con 0x7f79000aecc0 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 10 client.4995.objecter ms_dispatch 0x7f7900001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 3 client.4995.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 3 client.4995.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78feffd700 1 --2- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 0x7f78ec06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78feffd700 10 monclient: get_auth_request con 0x7f78ec068890 auth_method 0 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 20 client.4995.objecter dump_active .. 0 homeless 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78e6ffd700 10 client.4995.objecter ms_handle_connect 0x7f78ec068890 2023-09-30T01:08:40.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.376+0000 7f78feffd700 1 --2- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 0x7f78ec06ad40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f78f4007fd0 tx=0x7f78f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.380+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78f0053630 con 0x7f79000aecc0 2023-09-30T01:08:40.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.380+0000 7f78e6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.380+0000 7f78e6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:40.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f7905742700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:08:40.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f7905742700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:40.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f7905742700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:40.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7900072960 con 0x7f79000aecc0 2023-09-30T01:08:40.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f78e6ffd700 1 -- 172.21.15.115:0/1913653807 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f78f0056c50 con 0x7f79000aecc0 2023-09-30T01:08:40.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f78e6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:08:40.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.601+0000 7f78e6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:40.602 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:40.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 10 librados: watch_flush enter 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 10 librados: watch_flush exit 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 20 client.4995.objecter shutdown clearing up homeless session... 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 10 client.4995.objecter successfully canceled tick 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 msgr2=0x7f78ec06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 0x7f78ec06ad40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f78f4007fd0 tx=0x7f78f4009040 comp rx=0 tx=0).stop 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 10 monclient: shutdown 2023-09-30T01:08:40.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 msgr2=0x7f79000a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f78f00091e0 tx=0x7f78f0016000 comp rx=0 tx=0).stop 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 shutdown_connections 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> v2:172.21.15.115:6816/115635 conn(0x7f78ec068890 0x7f78ec06ad40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79001415c0 0x7f790013ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f79000ae8f0 0x7f790013e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 --2- 172.21.15.115:0/1913653807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79000aecc0 0x7f79000a99c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 >> 172.21.15.115:0/1913653807 conn(0x7f790006a460 msgr2=0x7f7900144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:40.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 shutdown_connections 2023-09-30T01:08:40.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 -- 172.21.15.115:0/1913653807 wait complete. 2023-09-30T01:08:40.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.603+0000 7f7905742700 1 librados: shutdown 2023-09-30T01:08:40.615 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T01:08:40.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:08:40.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 10 monclient: get_monmap_and_config 2023-09-30T01:08:40.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 10 monclient: build_initial_monmap 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 10 monclient: monmap: 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:40.732064+0000 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:40.732064+0000 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:40.732 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:40.733 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:40.733 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 1 Processor -- start 2023-09-30T01:08:40.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 1 -- start start 2023-09-30T01:08:40.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.731+0000 7f435b3c8700 10 monclient: init 2023-09-30T01:08:40.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:40.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-09-30T01:08:40.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 0x7f43540ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-b con 0x7f43540b1d70 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43540b2140 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-a con 0x7f43540a9b30 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43540aaba0 con 0x7f43540b2140 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43540ad070 con 0x7f43540a9b30 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43540ada00 con 0x7f43540b1d70 2023-09-30T01:08:40.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f435b3c8700 10 monclient(hunting): authenticate will time out at 1382.182469s 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 0x7f43540ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40318/0 (socket says 172.21.15.115:40318) 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 1 -- 172.21.15.115:0/417593972 learned_addr learned my addr 172.21.15.115:0/417593972 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): get_auth_request con 0x7f43540b1d70 auth_method 0 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): get_auth_request con 0x7f43540a9b30 auth_method 0 2023-09-30T01:08:40.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient(hunting): handle_auth_done global_id 4724 payload 274 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 -- 172.21.15.115:0/417593972 >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 msgr2=0x7f43540ad4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 --2- 172.21.15.115:0/417593972 >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 -- 172.21.15.115:0/417593972 >> v1:172.21.15.157:6789/0 conn(0x7f43540b2140 legacy=0x7f4354141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient: _finish_hunting 0 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 monclient: found mon.noname-b 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359164700 1 -- 172.21.15.115:0/417593972 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435406fb60 con 0x7f43540b1d70 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.732+0000 7f4359965700 1 --2- 172.21.15.115:0/417593972 >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f4359164700 1 --2- 172.21.15.115:0/417593972 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 0x7f43540ab870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4344009910 tx=0x7f43440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d39f5ecb3996f2e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 1 -- 172.21.15.115:0/417593972 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344011630 con 0x7f43540b1d70 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: dump: 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: _finish_auth 0 2023-09-30T01:08:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 1 -- 172.21.15.115:0/417593972 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344011790 con 0x7f43540b1d70 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 5 monclient: authenticate success, global_id 4724 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 1 -- 172.21.15.115:0/417593972 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344011b50 con 0x7f43540b1d70 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 4 set_mon_vals no callback set 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f434bfff700 10 monclient: dump: 2023-09-30T01:08:40.751 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.752 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 10 monclient: shutdown 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 -- 172.21.15.115:0/417593972 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 msgr2=0x7f43540ab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 --2- 172.21.15.115:0/417593972 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 0x7f43540ab870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4344009910 tx=0x7f43440091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 -- 172.21.15.115:0/417593972 shutdown_connections 2023-09-30T01:08:40.753 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 --2- 172.21.15.115:0/417593972 >> v2:172.21.15.115:3300/0 conn(0x7f43540a9b30 0x7f43540ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 --2- 172.21.15.115:0/417593972 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b1d70 0x7f43540ab870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 -- 172.21.15.115:0/417593972 >> 172.21.15.115:0/417593972 conn(0x7f4354068ce0 msgr2=0x7f43540b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 -- 172.21.15.115:0/417593972 shutdown_connections 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.733+0000 7f435b3c8700 1 -- 172.21.15.115:0/417593972 wait complete. 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 10 monclient: build_initial_monmap 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 10 monclient: monmap: 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:40.754 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 librados: starting msgr at 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 librados: starting objecter 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 Processor -- start 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 -- start start 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 librados: setting wanted keys 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 1 librados: calling monclient init 2023-09-30T01:08:40.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.734+0000 7f435b3c8700 10 monclient: init 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43540a9b30 0x7f4354150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-b con 0x7f43540a9b30 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 0x7f435413efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-a con 0x7f43540b1d70 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): picked mon.noname-c con 0x7f43540b2140 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435413f700 con 0x7f43540b1d70 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43541199a0 con 0x7f43540a9b30 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435413c480 con 0x7f43540b2140 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f435b3c8700 10 monclient(hunting): authenticate will time out at 1382.185554s 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 0x7f435413efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:49438/0 (socket says 172.21.15.115:49438) 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 -- 172.21.15.115:0/3061632771 learned_addr learned my addr 172.21.15.115:0/3061632771 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): get_auth_request con 0x7f43540b1d70 auth_method 0 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): get_auth_request con 0x7f43540b2140 auth_method 0 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43540a9b30 0x7f4354150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): get_auth_request con 0x7f43540a9b30 auth_method 0 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359164700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient(hunting): handle_auth_done global_id 4727 payload 824 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 -- 172.21.15.115:0/3061632771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43540a9b30 msgr2=0x7f4354150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43540a9b30 0x7f4354150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 -- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 msgr2=0x7f435413efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 0x7f435413efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient: _finish_hunting 0 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 monclient: found mon.noname-c 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4359965700 1 -- 172.21.15.115:0/3061632771 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c009ae0 con 0x7f43540b2140 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.735+0000 7f4358963700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 0x7f435413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f4359965700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f434c009fd0 tx=0x7f434c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 client.?.objecter ms_handle_connect 0x7f43540b2140 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c018070 con 0x7f43540b2140 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: dump: 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.761 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: _finish_auth 0 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 5 monclient: authenticate success, global_id 4727 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c00f040 con 0x7f43540b2140 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c00cc60 con 0x7f43540b2140 2023-09-30T01:08:40.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 10 monclient: _renew_subs 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4344009420 con 0x7f43540b2140 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f434b7fe700 4 set_mon_vals no callback set 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f43497fa700 10 monclient: dump: 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:40.763 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 10 monclient: _renew_subs 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354151c30 con 0x7f43540b2140 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.736+0000 7f435b3c8700 1 librados: init done 2023-09-30T01:08:40.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f435b3c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f435b3c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f435b3c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435413f8b0 con 0x7f43540b2140 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f434c003020 con 0x7f43540b2140 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f43497fa700 1 --2- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 0x7f433806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f4359164700 1 --2- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 0x7f433806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.737+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f434c084360 con 0x7f43540b2140 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f43497fa700 10 client.4727.objecter ms_dispatch 0x7f4354001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f43497fa700 3 client.4727.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f43497fa700 3 client.4727.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:40.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f4359164700 10 monclient: get_auth_request con 0x7f4338068890 auth_method 0 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f43497fa700 20 client.4727.objecter dump_active .. 0 homeless 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f43497fa700 10 client.4727.objecter ms_handle_connect 0x7f4338068890 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.738+0000 7f4359164700 1 --2- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 0x7f433806ad40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43440030c0 tx=0x7f4344016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.742+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f435413f8b0 con 0x7f43540b2140 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.742+0000 7f43497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:40.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.742+0000 7f43497fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:40.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f435b3c8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:40.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f435b3c8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:40.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f435b3c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:40.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f435413f870 con 0x7f43540b2140 2023-09-30T01:08:40.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f43497fa700 1 -- 172.21.15.115:0/3061632771 <== mon.2 v2:172.21.15.115:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11967 (secure 0 0 0) 0x7f434c051a50 con 0x7f43540b2140 2023-09-30T01:08:40.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f43497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:40.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.925+0000 7f43497fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:40.926 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:40.926 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:36.028196+0000","last_up_change":"2023-09-30T01:03:32.012648+0000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:03:32.045422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:03:33.941118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6801","nonce":115935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6803","nonce":115935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115935}]},"public_addr":"172.21.15.115:6800/115935","cluster_addr":"172.21.15.115:6801/115935","heartbeat_back_addr":"172.21.15.115:6803/115935","heartbeat_front_addr":"172.21.15.115:6802/115935","state":["exists","up"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":115950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6811","nonce":115950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115950}]},"public_addr":"172.21.15.115:6808/115950","cluster_addr":"172.21.15.115:6809/115950","heartbeat_back_addr":"172.21.15.115:6811/115950","heartbeat_front_addr":"172.21.15.115:6810/115950","state":["exists","up"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6805","nonce":115951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6807","nonce":115951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115951}]},"public_addr":"172.21.15.115:6804/115951","cluster_addr":"172.21.15.115:6805/115951","heartbeat_back_addr":"172.21.15.115:6807/115951","heartbeat_front_addr":"172.21.15.115:6806/115951","state":["exists","up"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6813","nonce":115952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":115952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115952}]},"public_addr":"172.21.15.115:6812/115952","cluster_addr":"172.21.15.115:6813/115952","heartbeat_back_addr":"172.21.15.115:6815/115952","heartbeat_front_addr":"172.21.15.115:6814/115952","state":["exists","up"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6801","nonce":115846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6803","nonce":115846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115846}]},"public_addr":"172.21.15.157:6800/115846","cluster_addr":"172.21.15.157:6801/115846","heartbeat_back_addr":"172.21.15.157:6803/115846","heartbeat_front_addr":"172.21.15.157:6802/115846","state":["exists","up"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6805","nonce":115847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":115847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115847}]},"public_addr":"172.21.15.157:6804/115847","cluster_addr":"172.21.15.157:6805/115847","heartbeat_back_addr":"172.21.15.157:6807/115847","heartbeat_front_addr":"172.21.15.157:6806/115847","state":["exists","up"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":115848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6811","nonce":115848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115848}]},"public_addr":"172.21.15.157:6808/115848","cluster_addr":"172.21.15.157:6809/115848","heartbeat_back_addr":"172.21.15.157:6811/115848","heartbeat_front_addr":"172.21.15.157:6810/115848","state":["exists","up"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6815","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115891}]},"public_addr":"172.21.15.157:6812/115891","cluster_addr":"172.21.15.157:6813/115891","heartbeat_back_addr":"172.21.15.157:6815/115891","heartbeat_front_addr":"172.21.15.157:6814/115891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.324933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.319648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.364624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.391390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.296016+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.336020+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.397884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.386449+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 10 librados: watch_flush enter 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 10 librados: watch_flush exit 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 20 client.4727.objecter shutdown clearing up homeless session... 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 10 client.4727.objecter successfully canceled tick 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 msgr2=0x7f433806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 0x7f433806ad40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f43440030c0 tx=0x7f4344016000 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 10 monclient: shutdown 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 msgr2=0x7f435413fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.927+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f434c009fd0 tx=0x7f434c00b040 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 shutdown_connections 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> v2:172.21.15.115:6816/115635 conn(0x7f4338068890 0x7f433806ad40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f43540b2140 0x7f435413fb90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43540a9b30 0x7f4354150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 --2- 172.21.15.115:0/3061632771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f43540b1d70 0x7f435413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 >> 172.21.15.115:0/3061632771 conn(0x7f4354068ce0 msgr2=0x7f43540addc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 shutdown_connections 2023-09-30T01:08:40.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 -- 172.21.15.115:0/3061632771 wait complete. 2023-09-30T01:08:40.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:40.928+0000 7f435b3c8700 1 librados: shutdown 2023-09-30T01:08:40.939 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:08:40.940 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:08:41.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.056+0000 7f6db6ba9700 10 monclient: get_monmap_and_config 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.056+0000 7f6db6ba9700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.056+0000 7f6db6ba9700 10 monclient: monmap: 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:41.057385+0000 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:41.057385+0000 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:41.058 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:41.058 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:41.058 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:41.058 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:41.058 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 Processor -- start 2023-09-30T01:08:41.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 -- start start 2023-09-30T01:08:41.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient: init 2023-09-30T01:08:41.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T01:08:41.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6db00ae8f0 0x7f6db00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-a con 0x7f6db00ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6db00aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6db01415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00a7040 con 0x7f6db01415c0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00a57f0 con 0x7f6db00aecc0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db00a5480 con 0x7f6db00ae8f0 2023-09-30T01:08:41.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db6ba9700 10 monclient(hunting): authenticate will time out at 1382.507788s 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db4945700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6db00ae8f0 0x7f6db00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db5146700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f6db01415c0 0x7f6db00adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:43282/0 (socket says 172.21.15.115:43282) 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db5146700 1 -- 172.21.15.115:0/488951542 learned_addr learned my addr 172.21.15.115:0/488951542 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db4945700 10 monclient(hunting): get_auth_request con 0x7f6db00ae8f0 auth_method 0 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db4945700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db4945700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.057+0000 7f6db4945700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient(hunting): handle_auth_done global_id 5004 payload 274 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 1 -- 172.21.15.115:0/488951542 >> v1:172.21.15.157:6789/0 conn(0x7f6db00aecc0 legacy=0x7f6db00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 1 -- 172.21.15.115:0/488951542 >> v1:172.21.15.115:6789/0 conn(0x7f6db01415c0 legacy=0x7f6db00adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 1 monclient: found mon.noname-a 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 1 -- 172.21.15.115:0/488951542 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718643196 0 0) 0x7f6db00a7040 con 0x7f6db01415c0 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 1 -- 172.21.15.115:0/488951542 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00a55c0 con 0x7f6db00ae8f0 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db4945700 1 --2- 172.21.15.115:0/488951542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00a5d10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6da40098e0 tx=0x7f6da40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944b5700eb4f0cf2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 1 -- 172.21.15.115:0/488951542 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4009e80 con 0x7f6db00ae8f0 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:41.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: dump: 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.077 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 1 -- 172.21.15.115:0/488951542 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da400b860 con 0x7f6db00ae8f0 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 5 monclient: authenticate success, global_id 5004 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 1 -- 172.21.15.115:0/488951542 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da400bbe0 con 0x7f6db00ae8f0 2023-09-30T01:08:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 4 set_mon_vals no callback set 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6daf7fe700 10 monclient: dump: 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.079 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 10 monclient: shutdown 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 1 -- 172.21.15.115:0/488951542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 msgr2=0x7f6db00a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.058+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/488951542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00a5d10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6da40098e0 tx=0x7f6da40091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 -- 172.21.15.115:0/488951542 shutdown_connections 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/488951542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00a5d10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 -- 172.21.15.115:0/488951542 >> 172.21.15.115:0/488951542 conn(0x7f6db006a460 msgr2=0x7f6db00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 -- 172.21.15.115:0/488951542 shutdown_connections 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 -- 172.21.15.115:0/488951542 wait complete. 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 10 monclient: monmap: 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:41.081 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 librados: starting msgr at 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 librados: starting objecter 2023-09-30T01:08:41.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.059+0000 7f6db6ba9700 1 Processor -- start 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 -- start start 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 librados: setting wanted keys 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 librados: calling monclient init 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient: init 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-a con 0x7f6db00ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6db00aecc0 0x7f6db013e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-c con 0x7f6db00aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 0x7f6db0151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): picked mon.noname-b con 0x7f6db01415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00a9e20 con 0x7f6db00ae8f0 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00ab4b0 con 0x7f6db01415c0 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00ab720 con 0x7f6db00aecc0 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db6ba9700 10 monclient(hunting): authenticate will time out at 1382.510707s 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:40356/0 (socket says 172.21.15.115:40356) 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 1 -- 172.21.15.115:0/1878221637 learned_addr learned my addr 172.21.15.115:0/1878221637 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): get_auth_request con 0x7f6db00ae8f0 auth_method 0 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6db00aecc0 0x7f6db013e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): get_auth_request con 0x7f6db00aecc0 auth_method 0 2023-09-30T01:08:41.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db4945700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db5146700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 0x7f6db0151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6daffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db5146700 10 monclient(hunting): get_auth_request con 0x7f6db01415c0 auth_method 0 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db5146700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db5146700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.060+0000 7f6db5146700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 10 monclient(hunting): handle_auth_done global_id 5010 payload 824 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 -- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6db00aecc0 msgr2=0x7f6db013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6db00aecc0 0x7f6db013e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 -- 172.21.15.115:0/1878221637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 msgr2=0x7f6db0151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 0x7f6db0151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 monclient: found mon.noname-a 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 -- 172.21.15.115:0/1878221637 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da4009420 con 0x7f6db00ae8f0 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db5146700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 0x7f6db0151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db4945700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6da400b3c0 tx=0x7f6da4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 client.?.objecter ms_handle_connect 0x7f6db00ae8f0 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da401f070 con 0x7f6db00ae8f0 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: dump: 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.089 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 5 monclient: authenticate success, global_id 5010 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da4000c90 con 0x7f6db00ae8f0 2023-09-30T01:08:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 10 monclient: _renew_subs 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da400bbe0 con 0x7f6db00ae8f0 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db013ee40 con 0x7f6db00ae8f0 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6daeffd700 4 set_mon_vals no callback set 2023-09-30T01:08:41.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6dacff9700 10 monclient: dump: 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.092 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 10 client.5010.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 10 monclient: _renew_subs 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db00a9740 con 0x7f6db00ae8f0 2023-09-30T01:08:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.061+0000 7f6db6ba9700 1 librados: init done 2023-09-30T01:08:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.062+0000 7f6db6ba9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.062+0000 7f6db6ba9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.062+0000 7f6db6ba9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.062+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db0072960 con 0x7f6db00ae8f0 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.062+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6da40118a0 con 0x7f6db00ae8f0 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 1 --2- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 0x7f6d9406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6daffff700 1 --2- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 0x7f6d9406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6da40852d0 con 0x7f6db00ae8f0 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 10 client.5010.objecter ms_dispatch 0x7f6db0001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 3 client.5010.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 3 client.5010.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6daffff700 10 monclient: get_auth_request con 0x7f6d94068890 auth_method 0 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 20 client.5010.objecter dump_active .. 0 homeless 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6dacff9700 10 client.5010.objecter ms_handle_connect 0x7f6d94068890 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.063+0000 7f6daffff700 1 --2- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 0x7f6d9406ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6db00ab090 tx=0x7f6d9c007ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.067+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6da4052b90 con 0x7f6db00ae8f0 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.067+0000 7f6dacff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.067+0000 7f6dacff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:41.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.245+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6db0141a40 con 0x7f6d94068890 2023-09-30T01:08:41.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.247+0000 7f6dacff9700 1 -- 172.21.15.115:0/1878221637 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72362 (secure 0 0 0) 0x7f6db0141a40 con 0x7f6d94068890 2023-09-30T01:08:41.250 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:41.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.249+0000 7f6db6ba9700 10 librados: watch_flush enter 2023-09-30T01:08:41.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.249+0000 7f6db6ba9700 10 librados: watch_flush exit 2023-09-30T01:08:41.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 20 client.5010.objecter shutdown clearing up homeless session... 2023-09-30T01:08:41.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 10 client.5010.objecter successfully canceled tick 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 msgr2=0x7f6d9406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 0x7f6d9406ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6db00ab090 tx=0x7f6d9c007ec0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 10 monclient: shutdown 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 msgr2=0x7f6db00aa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6da400b3c0 tx=0x7f6da4017000 comp rx=0 tx=0).stop 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 shutdown_connections 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> v2:172.21.15.115:6816/115635 conn(0x7f6d94068890 0x7f6d9406ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6db01415c0 0x7f6db0151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6db00ae8f0 0x7f6db00aa2b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 --2- 172.21.15.115:0/1878221637 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f6db00aecc0 0x7f6db013e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 >> 172.21.15.115:0/1878221637 conn(0x7f6db006a460 msgr2=0x7f6db0144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 shutdown_connections 2023-09-30T01:08:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 -- 172.21.15.115:0/1878221637 wait complete. 2023-09-30T01:08:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.250+0000 7f6db6ba9700 1 librados: shutdown 2023-09-30T01:08:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:08:41.261 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T01:08:40.064936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41924,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932796,"statfs":{"total":773094113280,"available":773051183104,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033682+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033682+0000","last_peered":"2023-09-30T01:03:35.033682+0000","last_clean":"2023-09-30T01:03:35.033682+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033682+0000","last_undegraded":"2023-09-30T01:03:35.033682+0000","last_fullsized":"2023-09-30T01:03:35.033682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.69399999999999995,"15min":0.84699999999999998},"min":{"1min":0.41099999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66300000000000003,"5min":6.883,"15min":6.883},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.70799999999999996},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.755,"15min":0.91400000000000003},"min":{"1min":0.46800000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.69799999999999995,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.71699999999999997,"15min":0.86099999999999999},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68200000000000005,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.70899999999999996,"15min":0.86699999999999999},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.70799999999999996,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.80800000000000005,"15min":1.0249999999999999},"min":{"1min":0.45700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.73799999999999999,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.84499999999999997,"15min":1.075},"min":{"1min":0.44,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60499999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48899999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.39100000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54600000000000004,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68999999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.373}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65400000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73099999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.441,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64700000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68799999999999994,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64600000000000002,"5min":1.103,"15min":1.103},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":1.014,"15min":1.014},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68300000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58799999999999997},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.70199999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64500000000000002,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.441}]},{"osd":2,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.38,"5min":0.42199999999999999,"15min":0.45300000000000001},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60699999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.65200000000000002,"5min":1.218,"15min":1.218},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67900000000000005,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76300000000000001,"5min":1.333,"15min":1.333},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.44700000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.433}]},{"osd":5,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.38100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.50700000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.04,"15min":1.04},"last":0.55300000000000005}]},{"osd":7,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.45100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90600000000000003,"5min":1.254,"15min":1.254},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.37,"15min":0.378},"min":{"1min":0.25800000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.53900000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.379,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55600000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.099,"15min":1.099},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79100000000000004,"5min":1.091,"15min":1.091},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86799999999999999,"5min":1.073,"15min":1.073},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92300000000000004,"5min":1.194,"15min":1.194},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85299999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.68600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":1.163,"15min":1.163},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65800000000000003,"15min":0.69199999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89500000000000002,"5min":1.393,"15min":1.393},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.52900000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.88400000000000001,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.504}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.371},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38800000000000001,"15min":0.38700000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67700000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63500000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.44800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81599999999999995,"5min":1.085,"15min":1.085},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.62},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749},{"interface":"front","average":{"1min":0.626,"5min":0.624,"15min":0.624},"min":{"1min":0.437,"5min":0.435,"15min":0.435},"max":{"1min":0.84799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.622,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89600000000000002,"5min":1.232,"15min":1.232},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.441,"15min":0.53500000000000003},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60199999999999998,"5min":5.476,"15min":5.476},"last":0.37},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.45900000000000002,"15min":0.53900000000000003},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51000000000000001,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.44}]},{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.439,"15min":0.47799999999999998},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50700000000000001,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.54100000000000004,"15min":0.65800000000000003},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.55100000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45600000000000002,"15min":0.497},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53500000000000003,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.53400000000000003,"15min":0.65000000000000002},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54200000000000004,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64800000000000002,"15min":0.67400000000000004},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81100000000000005,"5min":1.925,"15min":1.925},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89900000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.76900000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63300000000000001,"15min":0.68100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85099999999999998,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.70499999999999996,"15min":0.78500000000000003},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97199999999999998,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.78100000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":5.649,"15min":5.649},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63600000000000001,"15min":0.65800000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":1.855,"15min":1.855},"last":0.70599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79400000000000004,"5min":1.895,"15min":1.895},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66800000000000004,"15min":0.70999999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94899999999999995,"5min":3.294,"15min":3.294},"last":0.72099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.749,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.64400000000000002,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.72899999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.64600000000000002},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":1.387,"15min":1.387},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47199999999999998,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.59899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69899999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54800000000000004,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68899999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.432,"5min":0.38,"15min":0.38},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72699999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.505},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.498,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73299999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81899999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92100000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":1.014,"15min":1.014},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:08:41.265 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:08:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.381+0000 7f7d8f5aa700 10 monclient: get_monmap_and_config 2023-09-30T01:08:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.381+0000 7f7d8f5aa700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.381+0000 7f7d8f5aa700 10 monclient: monmap: 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:41.382579+0000 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:41.382579+0000 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:41.383 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 1 Processor -- start 2023-09-30T01:08:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 1 -- start start 2023-09-30T01:08:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient: init 2023-09-30T01:08:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d880ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d880aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.382+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d881415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880a7040 con 0x7f7d880ae8f0 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880a57f0 con 0x7f7d881415c0 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a5480 con 0x7f7d880aecc0 2023-09-30T01:08:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 10 monclient(hunting): authenticate will time out at 1382.833045s 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8d346700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f7d880ae8f0 0x7f7d880b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52772/0 (socket says 172.21.15.115:52772) 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36024/0 (socket says 172.21.15.115:36024) 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8d346700 1 -- 172.21.15.115:0/2799163591 learned_addr learned my addr 172.21.15.115:0/2799163591 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): get_auth_request con 0x7f7d880aecc0 auth_method 0 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient(hunting): handle_auth_done global_id 4733 payload 274 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 -- 172.21.15.115:0/2799163591 >> v1:172.21.15.157:6789/0 conn(0x7f7d881415c0 legacy=0x7f7d880adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 -- 172.21.15.115:0/2799163591 >> v1:172.21.15.115:6789/0 conn(0x7f7d880ae8f0 legacy=0x7f7d880b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 1 -- 172.21.15.115:0/2799163591 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123582618 0 0) 0x7f7d880a7040 con 0x7f7d880ae8f0 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 monclient: found mon.noname-b 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 -- 172.21.15.115:0/2799163591 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880a55c0 con 0x7f7d880aecc0 2023-09-30T01:08:41.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:41.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8cb45700 1 --2- 172.21.15.115:0/2799163591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009910 tx=0x7f7d780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1417021d94ee4e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 1 -- 172.21.15.115:0/2799163591 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800b860 con 0x7f7d880aecc0 2023-09-30T01:08:41.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: dump: 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.399 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.400 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.401 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 5 monclient: authenticate success, global_id 4733 2023-09-30T01:08:41.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 1 -- 172.21.15.115:0/2799163591 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7800b9c0 con 0x7f7d880aecc0 2023-09-30T01:08:41.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:41.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d8f5aa700 4 set_mon_vals no callback set 2023-09-30T01:08:41.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 1 -- 172.21.15.115:0/2799163591 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800bd60 con 0x7f7d880aecc0 2023-09-30T01:08:41.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.383+0000 7f7d7ffff700 10 monclient: dump: 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.403 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.405 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 10 monclient: shutdown 2023-09-30T01:08:41.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/2799163591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 msgr2=0x7f7d880a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/2799163591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009910 tx=0x7f7d780091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/2799163591 shutdown_connections 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/2799163591 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880aecc0 0x7f7d880a5d10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/2799163591 >> 172.21.15.115:0/2799163591 conn(0x7f7d8806a460 msgr2=0x7f7d880630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/2799163591 shutdown_connections 2023-09-30T01:08:41.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/2799163591 wait complete. 2023-09-30T01:08:41.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.384+0000 7f7d8f5aa700 10 monclient: monmap: 2023-09-30T01:08:41.407 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.408 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:41.408 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:41.408 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:41.408 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.408 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 librados: starting msgr at 2023-09-30T01:08:41.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 librados: starting objecter 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 Processor -- start 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 -- start start 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 librados: setting wanted keys 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 librados: calling monclient init 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient: init 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-c con 0x7f7d880ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d880aecc0 0x7f7d8814dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d880aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 0x7f7d880aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): picked mon.noname-a con 0x7f7d881415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a9de0 con 0x7f7d881415c0 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8814e280 con 0x7f7d880aecc0 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880aa7b0 con 0x7f7d880ae8f0 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.385+0000 7f7d8f5aa700 10 monclient(hunting): authenticate will time out at 1382.835984s 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36034/0 (socket says 172.21.15.115:36034) 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 0x7f7d880aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 -- 172.21.15.115:0/3790145343 learned_addr learned my addr 172.21.15.115:0/3790145343 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:41.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): get_auth_request con 0x7f7d880ae8f0 auth_method 0 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): get_auth_request con 0x7f7d881415c0 auth_method 0 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8cb45700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d880aecc0 0x7f7d8814dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8cb45700 10 monclient(hunting): get_auth_request con 0x7f7d880aecc0 auth_method 0 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8cb45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8cb45700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8cb45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient(hunting): handle_auth_done global_id 4736 payload 824 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 -- 172.21.15.115:0/3790145343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d880aecc0 msgr2=0x7f7d8814dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d880aecc0 0x7f7d8814dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 -- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 msgr2=0x7f7d880aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 0x7f7d880aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 monclient: found mon.noname-c 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 -- 172.21.15.115:0/3790145343 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d78009420 con 0x7f7d880ae8f0 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8db47700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 0x7f7d880aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8d346700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c840 tx=0x7f7d84007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 client.?.objecter ms_handle_connect 0x7f7d880ae8f0 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84005c00 con 0x7f7d880ae8f0 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: dump: 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.417 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.418 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.418 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.418 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.418 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.418 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 5 monclient: authenticate success, global_id 4736 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d84005d60 con 0x7f7d880ae8f0 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 10 monclient: _renew_subs 2023-09-30T01:08:41.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d880ad7a0 con 0x7f7d880ae8f0 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84007320 con 0x7f7d880ae8f0 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7d7fa700 10 monclient: dump: 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d7f7fe700 4 set_mon_vals no callback set 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 10 client.4736.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 10 monclient: _renew_subs 2023-09-30T01:08:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d880a9660 con 0x7f7d880ae8f0 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.386+0000 7f7d8f5aa700 1 librados: init done 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.387+0000 7f7d8f5aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.387+0000 7f7d8f5aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.387+0000 7f7d8f5aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.387+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88072960 con 0x7f7d880ae8f0 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7d8401b3b0 con 0x7f7d880ae8f0 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 1 --2- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 0x7f7d6c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7d84083430 con 0x7f7d880ae8f0 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 10 client.4736.objecter ms_dispatch 0x7f7d88001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 3 client.4736.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:41.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 3 client.4736.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d8cb45700 1 --2- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 0x7f7d6c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d8cb45700 10 monclient: get_auth_request con 0x7f7d6c068890 auth_method 0 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 20 client.4736.objecter dump_active .. 0 homeless 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d7d7fa700 10 client.4736.objecter ms_handle_connect 0x7f7d6c068890 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.388+0000 7f7d8cb45700 1 --2- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 0x7f7d6c06ad40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7d780093f0 tx=0x7f7d78017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.392+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d84050cf0 con 0x7f7d880ae8f0 2023-09-30T01:08:41.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.392+0000 7f7d7d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.392+0000 7f7d7d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:41.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.568+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d880ad990 con 0x7f7d6c068890 2023-09-30T01:08:41.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.571+0000 7f7d7d7fa700 1 -- 172.21.15.115:0/3790145343 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72362 (secure 0 0 0) 0x7f7d880ad990 con 0x7f7d6c068890 2023-09-30T01:08:41.573 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 10 librados: watch_flush enter 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 10 librados: watch_flush exit 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 20 client.4736.objecter shutdown clearing up homeless session... 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 10 client.4736.objecter successfully canceled tick 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 msgr2=0x7f7d6c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 0x7f7d6c06ad40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7d780093f0 tx=0x7f7d78017000 comp rx=0 tx=0).stop 2023-09-30T01:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 10 monclient: shutdown 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 msgr2=0x7f7d8813e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c840 tx=0x7f7d84007d90 comp rx=0 tx=0).stop 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 shutdown_connections 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> v2:172.21.15.115:6816/115635 conn(0x7f7d6c068890 0x7f7d6c06ad40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d881415c0 0x7f7d880aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f7d880ae8f0 0x7f7d8813e560 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 --2- 172.21.15.115:0/3790145343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d880aecc0 0x7f7d8814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 >> 172.21.15.115:0/3790145343 conn(0x7f7d8806a460 msgr2=0x7f7d88144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 shutdown_connections 2023-09-30T01:08:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 -- 172.21.15.115:0/3790145343 wait complete. 2023-09-30T01:08:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.573+0000 7f7d8f5aa700 1 librados: shutdown 2023-09-30T01:08:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:08:41.585 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T01:08:40.064936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41924,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932796,"statfs":{"total":773094113280,"available":773051183104,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033682+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033682+0000","last_peered":"2023-09-30T01:03:35.033682+0000","last_clean":"2023-09-30T01:03:35.033682+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033682+0000","last_undegraded":"2023-09-30T01:03:35.033682+0000","last_fullsized":"2023-09-30T01:03:35.033682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.69399999999999995,"15min":0.84699999999999998},"min":{"1min":0.41099999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66300000000000003,"5min":6.883,"15min":6.883},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.70799999999999996},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.755,"15min":0.91400000000000003},"min":{"1min":0.46800000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.69799999999999995,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.71699999999999997,"15min":0.86099999999999999},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68200000000000005,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.70899999999999996,"15min":0.86699999999999999},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.70799999999999996,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.80800000000000005,"15min":1.0249999999999999},"min":{"1min":0.45700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.73799999999999999,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.84499999999999997,"15min":1.075},"min":{"1min":0.44,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60499999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48899999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.39100000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54600000000000004,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68999999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.373}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65400000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73099999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.441,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64700000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68799999999999994,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64600000000000002,"5min":1.103,"15min":1.103},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":1.014,"15min":1.014},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68300000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58799999999999997},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.70199999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64500000000000002,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.441}]},{"osd":2,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.38,"5min":0.42199999999999999,"15min":0.45300000000000001},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60699999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.65200000000000002,"5min":1.218,"15min":1.218},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67900000000000005,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76300000000000001,"5min":1.333,"15min":1.333},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.44700000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.433}]},{"osd":5,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.38100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.50700000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.04,"15min":1.04},"last":0.55300000000000005}]},{"osd":7,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.45100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90600000000000003,"5min":1.254,"15min":1.254},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.37,"15min":0.378},"min":{"1min":0.25800000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.53900000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.379,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55600000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.099,"15min":1.099},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79100000000000004,"5min":1.091,"15min":1.091},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86799999999999999,"5min":1.073,"15min":1.073},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92300000000000004,"5min":1.194,"15min":1.194},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85299999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.68600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":1.163,"15min":1.163},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65800000000000003,"15min":0.69199999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89500000000000002,"5min":1.393,"15min":1.393},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.52900000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.88400000000000001,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.504}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.371},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38800000000000001,"15min":0.38700000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67700000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63500000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.44800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81599999999999995,"5min":1.085,"15min":1.085},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.62},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749},{"interface":"front","average":{"1min":0.626,"5min":0.624,"15min":0.624},"min":{"1min":0.437,"5min":0.435,"15min":0.435},"max":{"1min":0.84799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.622,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89600000000000002,"5min":1.232,"15min":1.232},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.441,"15min":0.53500000000000003},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60199999999999998,"5min":5.476,"15min":5.476},"last":0.37},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.45900000000000002,"15min":0.53900000000000003},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51000000000000001,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.44}]},{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.439,"15min":0.47799999999999998},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50700000000000001,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.54100000000000004,"15min":0.65800000000000003},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.55100000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45600000000000002,"15min":0.497},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53500000000000003,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.53400000000000003,"15min":0.65000000000000002},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54200000000000004,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64800000000000002,"15min":0.67400000000000004},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81100000000000005,"5min":1.925,"15min":1.925},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89900000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.76900000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63300000000000001,"15min":0.68100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85099999999999998,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.70499999999999996,"15min":0.78500000000000003},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97199999999999998,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.78100000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":5.649,"15min":5.649},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63600000000000001,"15min":0.65800000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":1.855,"15min":1.855},"last":0.70599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79400000000000004,"5min":1.895,"15min":1.895},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66800000000000004,"15min":0.70999999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94899999999999995,"5min":3.294,"15min":3.294},"last":0.72099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.749,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.64400000000000002,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.72899999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.64600000000000002},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":1.387,"15min":1.387},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47199999999999998,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.59899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69899999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54800000000000004,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68899999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.432,"5min":0.38,"15min":0.38},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72699999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.505},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.498,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73299999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81899999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92100000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":1.014,"15min":1.014},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:08:41.588 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:08:41.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:08:41.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.705+0000 7f125f393700 10 monclient: get_monmap_and_config 2023-09-30T01:08:41.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.705+0000 7f125f393700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.705+0000 7f125f393700 10 monclient: monmap: 2023-09-30T01:08:41.705 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:41.706363+0000 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:41.706363+0000 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:41.706 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:41.707 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:41.707 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:41.707 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:41.707 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.705+0000 7f125f393700 1 Processor -- start 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 -- start start 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient: init 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): picked mon.noname-b con 0x7f12580ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:41.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f12580aecc0 0x7f12580b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): picked mon.noname-c con 0x7f12580aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): picked mon.noname-a con 0x7f12581415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12580a7040 con 0x7f12581415c0 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12580b19b0 con 0x7f12580aecc0 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12580a57f0 con 0x7f12580ae8f0 2023-09-30T01:08:41.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.706+0000 7f125f393700 10 monclient(hunting): authenticate will time out at 1383.156910s 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56764/0 (socket says 172.21.15.115:56764) 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 1 -- 172.21.15.115:0/3381355066 learned_addr learned my addr 172.21.15.115:0/3381355066 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): get_auth_request con 0x7f12581415c0 auth_method 0 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36036/0 (socket says 172.21.15.115:36036) 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): get_auth_request con 0x7f12580ae8f0 auth_method 0 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125c92e700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.157:3300/0 conn(0x7f12580aecc0 0x7f12580b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient(hunting): handle_auth_done global_id 4739 payload 274 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 -- 172.21.15.115:0/3381355066 >> v2:172.21.15.157:3300/0 conn(0x7f12580aecc0 msgr2=0x7f12580b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.157:3300/0 conn(0x7f12580aecc0 0x7f12580b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 -- 172.21.15.115:0/3381355066 >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 msgr2=0x7f12580adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 monclient: found mon.noname-b 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 -- 172.21.15.115:0/3381355066 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580a5480 con 0x7f12580ae8f0 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d930700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125d12f700 1 --2- 172.21.15.115:0/3381355066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f125806a8b0 tx=0x7f12480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a79b5054edcbfa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 1 -- 172.21.15.115:0/3381355066 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248011630 con 0x7f12580ae8f0 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:41.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: dump: 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.722 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125f393700 5 monclient: authenticate success, global_id 4739 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 1 -- 172.21.15.115:0/3381355066 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248011790 con 0x7f12580ae8f0 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125f393700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 1 -- 172.21.15.115:0/3381355066 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248011b30 con 0x7f12580ae8f0 2023-09-30T01:08:41.723 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f125f393700 4 set_mon_vals no callback set 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.707+0000 7f124ffff700 10 monclient: dump: 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.724 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 10 monclient: shutdown 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 -- 172.21.15.115:0/3381355066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 msgr2=0x7f12580a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 --2- 172.21.15.115:0/3381355066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f125806a8b0 tx=0x7f12480091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 -- 172.21.15.115:0/3381355066 shutdown_connections 2023-09-30T01:08:41.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.115:3300/0 conn(0x7f12581415c0 0x7f12580adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 --2- 172.21.15.115:0/3381355066 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580ae8f0 0x7f12580a5d10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 --2- 172.21.15.115:0/3381355066 >> v2:172.21.15.157:3300/0 conn(0x7f12580aecc0 0x7f12580b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 -- 172.21.15.115:0/3381355066 >> 172.21.15.115:0/3381355066 conn(0x7f125806a460 msgr2=0x7f12580630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 -- 172.21.15.115:0/3381355066 shutdown_connections 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 -- 172.21.15.115:0/3381355066 wait complete. 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 10 monclient: build_initial_monmap 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 10 monclient: monmap: 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:41.726 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 librados: starting msgr at 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.708+0000 7f125f393700 1 librados: starting objecter 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 Processor -- start 2023-09-30T01:08:41.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 -- start start 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 librados: setting wanted keys 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 librados: calling monclient init 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient: init 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): picked mon.noname-a con 0x7f12580ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 0x7f1258140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): picked mon.noname-c con 0x7f12580aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:41.728 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12581415c0 0x7f1258151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): picked mon.noname-b con 0x7f12581415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12580a9f90 con 0x7f12580ae8f0 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12580ab340 con 0x7f12581415c0 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12580ab5b0 con 0x7f12580aecc0 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125f393700 10 monclient(hunting): authenticate will time out at 1383.159892s 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125c92e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 0x7f1258140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125d12f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.709+0000 7f125d12f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56768/0 (socket says 172.21.15.115:56768) 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): get_auth_request con 0x7f12580aecc0 auth_method 0 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 -- 172.21.15.115:0/1742684243 learned_addr learned my addr 172.21.15.115:0/1742684243 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): get_auth_request con 0x7f12580ae8f0 auth_method 0 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient(hunting): handle_auth_done global_id 5022 payload 824 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d930700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12581415c0 0x7f1258151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 -- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 msgr2=0x7f1258140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 0x7f1258140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 -- 172.21.15.115:0/1742684243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12581415c0 msgr2=0x7f1258151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12581415c0 0x7f1258151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient: _finish_hunting 0 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 monclient: found mon.noname-a 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 -- 172.21.15.115:0/1742684243 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1250009ae0 con 0x7f12580ae8f0 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125c92e700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 0x7f1258140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125d12f700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f12580a5930 tx=0x7f1248017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 client.?.objecter ms_handle_connect 0x7f12580ae8f0 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:41.732 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248011b10 con 0x7f12580ae8f0 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: dump: 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.733 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 5 monclient: authenticate success, global_id 5022 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12480233f0 con 0x7f12580ae8f0 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248023790 con 0x7f12580ae8f0 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:41.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124d7fa700 10 monclient: dump: 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f124f7fe700 4 set_mon_vals no callback set 2023-09-30T01:08:41.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 10 monclient: _renew_subs 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1248009420 con 0x7f12580ae8f0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 10 client.5022.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 10 monclient: _renew_subs 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12580a9700 con 0x7f12580ae8f0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.710+0000 7f125f393700 1 librados: init done 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.711+0000 7f125f393700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.711+0000 7f125f393700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.711+0000 7f125f393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.711+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12580a9980 con 0x7f12580ae8f0 2023-09-30T01:08:41.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.711+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1248023b40 con 0x7f12580ae8f0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 1 --2- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 0x7f123c06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1248085110 con 0x7f12580ae8f0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 10 client.5022.objecter ms_dispatch 0x7f1258001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f125c92e700 1 --2- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 0x7f123c06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 3 client.5022.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 3 client.5022.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f125c92e700 10 monclient: get_auth_request con 0x7f123c0688c0 auth_method 0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 20 client.5022.objecter dump_active .. 0 homeless 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f124d7fa700 10 client.5022.objecter ms_handle_connect 0x7f123c0688c0 2023-09-30T01:08:41.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.712+0000 7f125c92e700 1 --2- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 0x7f123c06ad70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f12580aaf20 tx=0x7f1254008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:41.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.716+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1248052a50 con 0x7f12580ae8f0 2023-09-30T01:08:41.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.716+0000 7f124d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:41.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.716+0000 7f124d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:41.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.895+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12580a9d60 con 0x7f123c0688c0 2023-09-30T01:08:41.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.898+0000 7f124d7fa700 1 -- 172.21.15.115:0/1742684243 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72362 (secure 0 0 0) 0x7f12580a9d60 con 0x7f123c0688c0 2023-09-30T01:08:41.899 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 10 librados: watch_flush enter 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 10 librados: watch_flush exit 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 20 client.5022.objecter shutdown clearing up homeless session... 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 10 client.5022.objecter successfully canceled tick 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 msgr2=0x7f123c06ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 0x7f123c06ad70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f12580aaf20 tx=0x7f1254008040 comp rx=0 tx=0).stop 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 10 monclient: shutdown 2023-09-30T01:08:41.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 msgr2=0x7f12580aa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f12580a5930 tx=0x7f1248017000 comp rx=0 tx=0).stop 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 shutdown_connections 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> v2:172.21.15.115:6816/115635 conn(0x7f123c0688c0 0x7f123c06ad70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12581415c0 0x7f1258151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f12580ae8f0 0x7f12580aa420 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 --2- 172.21.15.115:0/1742684243 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f12580aecc0 0x7f1258140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 >> 172.21.15.115:0/1742684243 conn(0x7f125806a460 msgr2=0x7f1258140b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 shutdown_connections 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 -- 172.21.15.115:0/1742684243 wait complete. 2023-09-30T01:08:41.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:41.900+0000 7f125f393700 1 librados: shutdown 2023-09-30T01:08:41.903 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:08:41.912 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T01:08:40.064936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41924,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932796,"statfs":{"total":773094113280,"available":773051183104,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033682+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033682+0000","last_peered":"2023-09-30T01:03:35.033682+0000","last_clean":"2023-09-30T01:03:35.033682+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033682+0000","last_undegraded":"2023-09-30T01:03:35.033682+0000","last_fullsized":"2023-09-30T01:03:35.033682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.69399999999999995,"15min":0.84699999999999998},"min":{"1min":0.41099999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66300000000000003,"5min":6.883,"15min":6.883},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.62,"15min":0.70799999999999996},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.755,"15min":0.91400000000000003},"min":{"1min":0.46800000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.69799999999999995,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.71699999999999997,"15min":0.86099999999999999},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68200000000000005,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.70899999999999996,"15min":0.86699999999999999},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.70799999999999996,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.80800000000000005,"15min":1.0249999999999999},"min":{"1min":0.45700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.73799999999999999,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.84499999999999997,"15min":1.075},"min":{"1min":0.44,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60499999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48899999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.39100000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54600000000000004,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68999999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.373}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65400000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73099999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.441,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64700000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68799999999999994,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64600000000000002,"5min":1.103,"15min":1.103},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":1.014,"15min":1.014},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68300000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58799999999999997},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.70199999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64500000000000002,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.441}]},{"osd":2,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.38,"5min":0.42199999999999999,"15min":0.45300000000000001},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60699999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.65200000000000002,"5min":1.218,"15min":1.218},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67900000000000005,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.38800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76300000000000001,"5min":1.333,"15min":1.333},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.44700000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.433}]},{"osd":5,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.38100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.50700000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.04,"15min":1.04},"last":0.55300000000000005}]},{"osd":7,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.45100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90600000000000003,"5min":1.254,"15min":1.254},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.37,"15min":0.378},"min":{"1min":0.25800000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.53900000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.251},{"interface":"front","average":{"1min":0.379,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52700000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55600000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.099,"15min":1.099},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79100000000000004,"5min":1.091,"15min":1.091},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66600000000000004,"15min":0.66900000000000004},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86799999999999999,"5min":1.073,"15min":1.073},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.92300000000000004,"5min":1.194,"15min":1.194},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85299999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.68600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":1.163,"15min":1.163},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65800000000000003,"15min":0.69199999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89500000000000002,"5min":1.393,"15min":1.393},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68400000000000005,"15min":0.70199999999999996},"min":{"1min":0.52900000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.88400000000000001,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.504}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.371},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38800000000000001,"15min":0.38700000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67700000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63500000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.44800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81599999999999995,"5min":1.085,"15min":1.085},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.62},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749},{"interface":"front","average":{"1min":0.626,"5min":0.624,"15min":0.624},"min":{"1min":0.437,"5min":0.435,"15min":0.435},"max":{"1min":0.84799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.622,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89600000000000002,"5min":1.232,"15min":1.232},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.441,"15min":0.53500000000000003},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60199999999999998,"5min":5.476,"15min":5.476},"last":0.37},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.45900000000000002,"15min":0.53900000000000003},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51000000000000001,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.44}]},{"osd":1,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.439,"15min":0.47799999999999998},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50700000000000001,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.54100000000000004,"15min":0.65800000000000003},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.55100000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45600000000000002,"15min":0.497},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53500000000000003,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.53400000000000003,"15min":0.65000000000000002},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54200000000000004,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64800000000000002,"15min":0.67400000000000004},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81100000000000005,"5min":1.925,"15min":1.925},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.72899999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89900000000000002,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.76900000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63300000000000001,"15min":0.68100000000000005},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85099999999999998,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.70499999999999996,"15min":0.78500000000000003},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97199999999999998,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.78100000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":5.649,"15min":5.649},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63600000000000001,"15min":0.65800000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":1.855,"15min":1.855},"last":0.70599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79400000000000004,"5min":1.895,"15min":1.895},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66800000000000004,"15min":0.70999999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94899999999999995,"5min":3.294,"15min":3.294},"last":0.72099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.749,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.64400000000000002,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.72899999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.60099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.64600000000000002},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":1.387,"15min":1.387},"last":0.61299999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47199999999999998,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.59899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69899999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54800000000000004,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68899999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.47799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.432,"5min":0.38,"15min":0.38},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72699999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.505},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52700000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.498,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73299999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81899999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92100000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":1.014,"15min":1.014},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:08:41.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:08:42.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.031+0000 7f289f3cf700 10 monclient: get_monmap_and_config 2023-09-30T01:08:42.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.031+0000 7f289f3cf700 10 monclient: build_initial_monmap 2023-09-30T01:08:42.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.031+0000 7f289f3cf700 10 monclient: monmap: 2023-09-30T01:08:42.031 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:42.032387+0000 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:42.032387+0000 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:42.032 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 Processor -- start 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 -- start start 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient: init 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T01:08:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28980ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f28980aecc0 0x7f28980b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-a con 0x7f28980aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f28980a5c80 0x7f28980adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-c con 0x7f28980a5c80 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28980a7040 con 0x7f28980ae8f0 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28980a5480 con 0x7f28980aecc0 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28980a5760 con 0x7f28980a5c80 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289f3cf700 10 monclient(hunting): authenticate will time out at 1383.482873s 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289c96a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f28980aecc0 0x7f28980b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.032+0000 7f289c96a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f28980aecc0 0x7f28980b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56770/0 (socket says 172.21.15.115:56770) 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 -- 172.21.15.115:0/2078526458 learned_addr learned my addr 172.21.15.115:0/2078526458 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): get_auth_request con 0x7f28980aecc0 auth_method 0 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:42.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 1 --2- 172.21.15.115:0/2078526458 >> v2:172.21.15.157:3300/0 conn(0x7f28980a5c80 0x7f28980adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): get_auth_request con 0x7f28980a5c80 auth_method 0 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:42.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289d96c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient(hunting): handle_auth_done global_id 5025 payload 274 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 -- 172.21.15.115:0/2078526458 >> v2:172.21.15.157:3300/0 conn(0x7f28980a5c80 msgr2=0x7f28980adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 --2- 172.21.15.115:0/2078526458 >> v2:172.21.15.157:3300/0 conn(0x7f28980a5c80 0x7f28980adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 -- 172.21.15.115:0/2078526458 >> v1:172.21.15.157:6789/0 conn(0x7f28980ae8f0 legacy=0x7f28981415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient: _finish_hunting 0 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 monclient: found mon.noname-a 2023-09-30T01:08:42.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 1 -- 172.21.15.115:0/2078526458 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546900723 0 0) 0x7f28980a7040 con 0x7f28980ae8f0 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 -- 172.21.15.115:0/2078526458 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28980b1970 con 0x7f28980aecc0 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289c96a700 1 --2- 172.21.15.115:0/2078526458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980aecc0 0x7f28980b1430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2898141cd0 tx=0x7f2894009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8624fb07b361b06 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 1 -- 172.21.15.115:0/2078526458 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894009e70 con 0x7f28980aecc0 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: dump: 2023-09-30T01:08:42.048 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: _finish_auth 0 2023-09-30T01:08:42.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:42.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 1 -- 172.21.15.115:0/2078526458 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289400b860 con 0x7f28980aecc0 2023-09-30T01:08:42.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289f3cf700 5 monclient: authenticate success, global_id 5025 2023-09-30T01:08:42.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289f3cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289f3cf700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 1 -- 172.21.15.115:0/2078526458 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289400bc00 con 0x7f28980aecc0 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f289f3cf700 4 set_mon_vals no callback set 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.033+0000 7f288ffff700 10 monclient: dump: 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:42.051 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 10 monclient: shutdown 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:42.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 -- 172.21.15.115:0/2078526458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980aecc0 msgr2=0x7f28980b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 --2- 172.21.15.115:0/2078526458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980aecc0 0x7f28980b1430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2898141cd0 tx=0x7f2894009250 comp rx=0 tx=0).stop 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 -- 172.21.15.115:0/2078526458 shutdown_connections 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 --2- 172.21.15.115:0/2078526458 >> v2:172.21.15.157:3300/0 conn(0x7f28980a5c80 0x7f28980adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 --2- 172.21.15.115:0/2078526458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980aecc0 0x7f28980b1430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 -- 172.21.15.115:0/2078526458 >> 172.21.15.115:0/2078526458 conn(0x7f289806a490 msgr2=0x7f28980630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 -- 172.21.15.115:0/2078526458 shutdown_connections 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 1 -- 172.21.15.115:0/2078526458 wait complete. 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 10 monclient: build_initial_monmap 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.034+0000 7f289f3cf700 10 monclient: monmap: 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:42.053 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 librados: starting msgr at 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 librados: starting objecter 2023-09-30T01:08:42.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 Processor -- start 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 -- start start 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 librados: setting wanted keys 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 1 librados: calling monclient init 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.035+0000 7f289f3cf700 10 monclient: init 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f28980a9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-c con 0x7f28980a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-a con 0x7f28980ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f289814e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): picked mon.noname-b con 0x7f28980aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28980a9f80 con 0x7f28980ae8f0 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28980a9c80 con 0x7f28980aecc0 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289813d720 con 0x7f28980a5c80 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289f3cf700 10 monclient(hunting): authenticate will time out at 1383.486475s 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f28980a9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f28980a9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36054/0 (socket says 172.21.15.115:36054) 2023-09-30T01:08:42.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 1 -- 172.21.15.115:0/889436154 learned_addr learned my addr 172.21.15.115:0/889436154 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): get_auth_request con 0x7f28980a5c80 auth_method 0 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): get_auth_request con 0x7f28980ae8f0 auth_method 0 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:42.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient(hunting): handle_auth_done global_id 5028 payload 824 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 msgr2=0x7f28980a9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f28980a9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 msgr2=0x7f289814e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d96c700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f289814e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f289814e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient: _finish_hunting 0 2023-09-30T01:08:42.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 monclient: found mon.noname-a 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289c96a700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28940094f0 con 0x7f28980ae8f0 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.036+0000 7f289d16b700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f28980a9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2894013000 tx=0x7f2894017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 client.?.objecter ms_handle_connect 0x7f28980ae8f0 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289400bbe0 con 0x7f28980ae8f0 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: dump: 2023-09-30T01:08:42.059 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:08:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 5 monclient: authenticate success, global_id 5028 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 10 monclient: _renew_subs 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2894011420 con 0x7f28980ae8f0 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289814bb10 con 0x7f28980ae8f0 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28940117a0 con 0x7f28980ae8f0 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: dump: 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:42.061 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288f7fe700 4 set_mon_vals no callback set 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 10 client.5028.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 10 monclient: _renew_subs 2023-09-30T01:08:42.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289814e900 con 0x7f28980ae8f0 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289f3cf700 1 librados: init done 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289c96a700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 msgr2=0x7f28980a9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289c96a700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 msgr2=0x7f28980a9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2894013000 tx=0x7f2894017000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2894013000 tx=0x7f2894017000 comp rx=0 tx=0).stop 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: ms_handle_reset current mon [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 msgr2=0x7f28980a9740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f28980ae8f0 0x7f28980a9740 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:42.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 0x7f287c0075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): picked mon.a con 0x7f287c0051f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f287c00a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): picked mon.b con 0x7f28980aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f287c00ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): picked mon.c con 0x7f28980a5c80 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c00d3c0 con 0x7f287c0051f0 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c00d530 con 0x7f28980aecc0 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:42.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c00d6a0 con 0x7f28980a5c80 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 0x7f287c0075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): get_auth_request con 0x7f287c0051f0 auth_method 0 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient(hunting): handle_auth_done global_id 5028 payload 293 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 msgr2=0x7f287c00ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f287c00ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 msgr2=0x7f287c00a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f287c00a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient: _finish_hunting 0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 monclient: found mon.a 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f287c00d990 con 0x7f287c0051f0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 10 monclient: _finish_auth 0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 client.5028.objecter ms_handle_connect 0x7f287c0051f0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f289d16b700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 0x7f287c0075d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2888007fd0 tx=0x7f2888009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:42.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.037+0000 7f288d7fa700 10 client.5028.objecter resend_mon_ops 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288800faa0 con 0x7f287c0051f0 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 10 monclient: dump: 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:42.067 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288800fc00 con 0x7f287c0051f0 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f288effd700 4 set_mon_vals no callback set 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f289f3cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f289f3cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f289f3cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.038+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28700050b0 con 0x7f287c0051f0 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f288800fd70 con 0x7f287c0051f0 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 1 --2- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 0x7f287c073610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:42.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 0x7f287c073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 4 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f28880817a0 con 0x7f287c0051f0 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f289c96a700 10 monclient: get_auth_request con 0x7f287c071160 auth_method 0 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 10 client.5028.objecter ms_dispatch 0x7f2898001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 3 client.5028.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 3 client.5028.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.039+0000 7f288d7fa700 20 client.5028.objecter dump_active .. 0 homeless 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.040+0000 7f288d7fa700 10 client.5028.objecter ms_handle_connect 0x7f287c071160 2023-09-30T01:08:42.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.040+0000 7f289c96a700 1 --2- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 0x7f287c073610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f28980a7930 tx=0x7f289400b050 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:42.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.043+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f288804f110 con 0x7f287c0051f0 2023-09-30T01:08:42.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.043+0000 7f288d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:42.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.043+0000 7f288d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:42.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.223+0000 7f289f3cf700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:42.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.223+0000 7f289f3cf700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:42.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.223+0000 7f289f3cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:42.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.223+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2870005630 con 0x7f287c0051f0 2023-09-30T01:08:42.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.224+0000 7f288d7fa700 1 -- 172.21.15.115:0/889436154 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11967 (secure 0 0 0) 0x7f2888052730 con 0x7f287c0051f0 2023-09-30T01:08:42.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.224+0000 7f288d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:08:42.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.224+0000 7f288d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:08:42.225 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:42.225 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"fsid":"c6b635fb-787f-4de6-b95e-9004242e56e2","created":"2023-09-30T01:03:22.558117+0000","modified":"2023-09-30T01:03:36.028196+0000","last_up_change":"2023-09-30T01:03:32.012648+0000","last_in_change":"2023-09-30T01:03:25.192334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:03:32.045422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:03:33.941118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f8068422-b1d0-4236-a4eb-fa8aecefe228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6801","nonce":115935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6803","nonce":115935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115935}]},"public_addr":"172.21.15.115:6800/115935","cluster_addr":"172.21.15.115:6801/115935","heartbeat_back_addr":"172.21.15.115:6803/115935","heartbeat_front_addr":"172.21.15.115:6802/115935","state":["exists","up"]},{"osd":1,"uuid":"e73989ad-7372-4777-9fa6-c96c8b3bd763","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":115950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6811","nonce":115950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115950}]},"public_addr":"172.21.15.115:6808/115950","cluster_addr":"172.21.15.115:6809/115950","heartbeat_back_addr":"172.21.15.115:6811/115950","heartbeat_front_addr":"172.21.15.115:6810/115950","state":["exists","up"]},{"osd":2,"uuid":"d8599e95-3db1-402a-a597-42d388a61a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6805","nonce":115951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6807","nonce":115951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115951}]},"public_addr":"172.21.15.115:6804/115951","cluster_addr":"172.21.15.115:6805/115951","heartbeat_back_addr":"172.21.15.115:6807/115951","heartbeat_front_addr":"172.21.15.115:6806/115951","state":["exists","up"]},{"osd":3,"uuid":"08039bd8-9791-4ea6-a7d6-638fa734cdaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6813","nonce":115952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":115952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115952}]},"public_addr":"172.21.15.115:6812/115952","cluster_addr":"172.21.15.115:6813/115952","heartbeat_back_addr":"172.21.15.115:6815/115952","heartbeat_front_addr":"172.21.15.115:6814/115952","state":["exists","up"]},{"osd":4,"uuid":"38c57cc5-98c8-4d46-913a-a119cf08ed58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6801","nonce":115846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6803","nonce":115846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115846}]},"public_addr":"172.21.15.157:6800/115846","cluster_addr":"172.21.15.157:6801/115846","heartbeat_back_addr":"172.21.15.157:6803/115846","heartbeat_front_addr":"172.21.15.157:6802/115846","state":["exists","up"]},{"osd":5,"uuid":"fae1ca7c-8b05-406f-bfa7-c61093a32111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6805","nonce":115847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6807","nonce":115847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115847}]},"public_addr":"172.21.15.157:6804/115847","cluster_addr":"172.21.15.157:6805/115847","heartbeat_back_addr":"172.21.15.157:6807/115847","heartbeat_front_addr":"172.21.15.157:6806/115847","state":["exists","up"]},{"osd":6,"uuid":"5a22a721-1492-4746-9b2f-c28fd52c465f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":115848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6811","nonce":115848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115848}]},"public_addr":"172.21.15.157:6808/115848","cluster_addr":"172.21.15.157:6809/115848","heartbeat_back_addr":"172.21.15.157:6811/115848","heartbeat_front_addr":"172.21.15.157:6810/115848","state":["exists","up"]},{"osd":7,"uuid":"2963067c-f278-4458-a7b2-9163558524b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6815","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115891}]},"public_addr":"172.21.15.157:6812/115891","cluster_addr":"172.21.15.157:6813/115891","heartbeat_back_addr":"172.21.15.157:6815/115891","heartbeat_front_addr":"172.21.15.157:6814/115891","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.324933+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.319648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.364624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.391390+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.296016+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.336020+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.397884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:03:30.386449+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 10 librados: watch_flush enter 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 10 librados: watch_flush exit 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 20 client.5028.objecter shutdown clearing up homeless session... 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 10 client.5028.objecter successfully canceled tick 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 msgr2=0x7f287c073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 0x7f287c073610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f28980a7930 tx=0x7f289400b050 comp rx=0 tx=0).stop 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 10 monclient: shutdown 2023-09-30T01:08:42.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 msgr2=0x7f287c0075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 0x7f287c0075d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2888007fd0 tx=0x7f2888009040 comp rx=0 tx=0).stop 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 shutdown_connections 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> v2:172.21.15.115:6816/115635 conn(0x7f287c071160 0x7f287c073610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f28980a5c80 0x7f287c00ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28980aecc0 0x7f287c00a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 --2- 172.21.15.115:0/889436154 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f287c0051f0 0x7f287c0075d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 >> 172.21.15.115:0/889436154 conn(0x7f289806a490 msgr2=0x7f2898070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 shutdown_connections 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 -- 172.21.15.115:0/889436154 wait complete. 2023-09-30T01:08:42.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:42.226+0000 7f289f3cf700 1 librados: shutdown 2023-09-30T01:08:43.238 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T01:08:43.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:43.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.372+0000 7fcfc2b30700 10 monclient: get_monmap_and_config 2023-09-30T01:08:43.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.372+0000 7fcfc2b30700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.372+0000 7fcfc2b30700 10 monclient: monmap: 2023-09-30T01:08:43.372 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.372 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:43.373239+0000 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:43.373239+0000 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:43.373 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 Processor -- start 2023-09-30T01:08:43.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 -- start start 2023-09-30T01:08:43.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient: init 2023-09-30T01:08:43.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfbc0a5640 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-b con 0x7fcfbc13e040 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 0x7fcfbc13b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-c con 0x7fcfbc13acd0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfbc0a8220 con 0x7fcfbc0a5640 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfbc0a6dc0 con 0x7fcfbc13acd0 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0a7d20 con 0x7fcfbc13e040 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfc2b30700 10 monclient(hunting): authenticate will time out at 1384.823675s 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36064/0 (socket says 172.21.15.115:36064) 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 1 -- 172.21.15.115:0/41529403 learned_addr learned my addr 172.21.15.115:0/41529403 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): get_auth_request con 0x7fcfbc13e040 auth_method 0 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.373+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/41529403 >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 0x7fcfbc13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_done global_id 4748 payload 274 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 -- 172.21.15.115:0/41529403 >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 msgr2=0x7fcfbc13b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/41529403 >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 0x7fcfbc13b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 -- 172.21.15.115:0/41529403 >> v1:172.21.15.157:6789/0 conn(0x7fcfbc0a5640 legacy=0x7fcfbc0a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 monclient: found mon.noname-b 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 -- 172.21.15.115:0/41529403 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc0a5a10 con 0x7fcfbc13e040 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc10cd700 10 monclient: get_auth_request con 0x7fcfbc13acd0 auth_method 0 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/41529403 >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 0x7fcfbc13b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:08:43.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/41529403 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80099b0 tx=0x7fcfa8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eba2411a2450af8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 1 -- 172.21.15.115:0/41529403 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800b820 con 0x7fcfbc13e040 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: dump: 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.387 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 1 -- 172.21.15.115:0/41529403 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa800b980 con 0x7fcfbc13e040 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 5 monclient: authenticate success, global_id 4748 2023-09-30T01:08:43.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 1 -- 172.21.15.115:0/41529403 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800bd20 con 0x7fcfbc13e040 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 4 set_mon_vals no callback set 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfbb7fe700 10 monclient: dump: 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.389 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 10 monclient: shutdown 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 -- 172.21.15.115:0/41529403 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 msgr2=0x7fcfbc13e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/41529403 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80099b0 tx=0x7fcfa8009250 comp rx=0 tx=0).stop 2023-09-30T01:08:43.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 -- 172.21.15.115:0/41529403 shutdown_connections 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/41529403 >> v2:172.21.15.157:3300/0 conn(0x7fcfbc13acd0 0x7fcfbc13b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/41529403 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc13e040 0x7fcfbc13e4c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 -- 172.21.15.115:0/41529403 >> 172.21.15.115:0/41529403 conn(0x7fcfbc068e40 msgr2=0x7fcfbc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.374+0000 7fcfc2b30700 1 -- 172.21.15.115:0/41529403 shutdown_connections 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 1 -- 172.21.15.115:0/41529403 wait complete. 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 10 monclient: monmap: 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:43.392 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:43.393 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:43.393 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 1 librados: starting msgr at 2023-09-30T01:08:43.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 1 librados: starting objecter 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.375+0000 7fcfc2b30700 1 Processor -- start 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 -- start start 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 librados: setting wanted keys 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 librados: calling monclient init 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient: init 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 0x7fcfbc0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-c con 0x7fcfbc0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfbc13acd0 0x7fcfbc0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-b con 0x7fcfbc13acd0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfbc13e040 0x7fcfbc0a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): picked mon.noname-a con 0x7fcfbc13e040 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0abbb0 con 0x7fcfbc13e040 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0b3570 con 0x7fcfbc13acd0 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0b3ea0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc2b30700 10 monclient(hunting): authenticate will time out at 1384.826641s 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 0x7fcfbc0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 0x7fcfbc0b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36068/0 (socket says 172.21.15.115:36068) 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 1 -- 172.21.15.115:0/3056238039 learned_addr learned my addr 172.21.15.115:0/3056238039 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): get_auth_request con 0x7fcfbc0a5640 auth_method 0 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfbc13e040 0x7fcfbc0a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): get_auth_request con 0x7fcfbc13e040 auth_method 0 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc08cc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfbc13acd0 0x7fcfbc0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfc10cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfbbfff700 10 monclient(hunting): get_auth_request con 0x7fcfbc13acd0 auth_method 0 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfbbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfbbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.376+0000 7fcfbbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfbbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 10 monclient(hunting): handle_auth_done global_id 4751 payload 824 2023-09-30T01:08:43.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 -- 172.21.15.115:0/3056238039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfbc13acd0 msgr2=0x7fcfbc0b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfbc13acd0 0x7fcfbc0b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 -- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfbc13e040 msgr2=0x7fcfbc0a9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfbc13e040 0x7fcfbc0a9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 monclient: found mon.noname-c 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 -- 172.21.15.115:0/3056238039 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80094f0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcfbc13e040 0x7fcfbc0a9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc08cc700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 0x7fcfbc0b3030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c5d0 tx=0x7fcfb0007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 client.?.objecter ms_handle_connect 0x7fcfbc0a5640 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011590 con 0x7fcfbc0a5640 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: dump: 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.400 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 5 monclient: authenticate success, global_id 4751 2023-09-30T01:08:43.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb00116f0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 10 monclient: _renew_subs 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011ab0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc0b24b0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfbaffd700 4 set_mon_vals no callback set 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfb8ff9700 10 monclient: dump: 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.402 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 10 client.4751.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 10 monclient: _renew_subs 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc0a9dd0 con 0x7fcfbc0a5640 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.377+0000 7fcfc2b30700 1 librados: init done 2023-09-30T01:08:43.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 10 client.4751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 15 client.4751.objecter _session_command_op_assign -1 1 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 20 client.4751.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 10 client.4751.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfbc072960 con 0x7fcfbc0a5640 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.378+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcfb001c460 con 0x7fcfbc0a5640 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6816/115635 conn(0x7fcfa0068890 0x7fcfa006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcfb0083b70 con 0x7fcfbc0a5640 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 client.4751.objecter ms_dispatch 0x7fcfbc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 3 client.4751.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 3 client.4751.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:43.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6816/115635 conn(0x7fcfa0068890 0x7fcfa006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfbbfff700 10 monclient: get_auth_request con 0x7fcfa0068890 auth_method 0 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 client.4751.objecter checking command 1 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6800/115935 conn(0x7fcfa006e800 0x7fcfa0070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter _get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 15 client.4751.objecter _session_command_op_remove -1 1 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter _get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 15 client.4751.objecter _session_command_op_assign 0 1 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 4 2023-09-30T01:08:43.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 client.4751.objecter _send_command 1 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 --> v2:172.21.15.115:6800/115935 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfa0071280 con 0x7fcfa006e800 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 20 client.4751.objecter dump_active .. 0 homeless 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb0088240 con 0x7fcfbc0a5640 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6800/115935 conn(0x7fcfa006e800 0x7fcfa0070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfc10cd700 10 monclient: get_auth_request con 0x7fcfa006e800 auth_method 0 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 client.4751.objecter ms_handle_connect 0x7fcfa0068890 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6816/115635 conn(0x7fcfa0068890 0x7fcfa006ad40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80099b0 tx=0x7fcfa8016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfb8ff9700 10 client.4751.objecter ms_handle_connect 0x7fcfa006e800 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.379+0000 7fcfc10cd700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6800/115935 conn(0x7fcfa006e800 0x7fcfa0070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.380+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== osd.0 v2:172.21.15.115:6800/115935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fcfa0071280 con 0x7fcfa006e800 2023-09-30T01:08:43.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.380+0000 7fcfb8ff9700 10 client.4751.objecter ms_dispatch 0x7fcfbc001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:43.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.380+0000 7fcfb8ff9700 10 client.4751.objecter _finish_command 1 = system:0 2023-09-30T01:08:43.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.380+0000 7fcfb8ff9700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 4 2023-09-30T01:08:43.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.380+0000 7fcfb8ff9700 15 client.4751.objecter _session_command_op_remove 0 1 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 10 client.4751.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 15 client.4751.objecter _session_command_op_assign -1 2 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 20 client.4751.objecter _get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 20 client.4751.objecter _get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 15 client.4751.objecter _session_command_op_remove -1 2 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 20 client.4751.objecter get_session s=0x7fcfa006e150 osd=0 3 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 15 client.4751.objecter _session_command_op_assign 0 2 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 4 2023-09-30T01:08:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 10 client.4751.objecter _send_command 2 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.393+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 --> v2:172.21.15.115:6800/115935 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcfbc0aec70 con 0x7fcfa006e800 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfb8ff9700 1 -- 172.21.15.115:0/3056238039 <== osd.0 v2:172.21.15.115:6800/115935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fcfbc0aec70 con 0x7fcfa006e800 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfb8ff9700 10 client.4751.objecter ms_dispatch 0x7fcfbc001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfb8ff9700 10 client.4751.objecter _finish_command 2 = system:0 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfb8ff9700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 4 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfb8ff9700 15 client.4751.objecter _session_command_op_remove 0 2 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:43.409 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:43.410 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:43.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 10 librados: watch_flush enter 2023-09-30T01:08:43.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 10 librados: watch_flush exit 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 10 client.4751.objecter close_session for osd.0 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6800/115935 conn(0x7fcfa006e800 msgr2=0x7fcfa0070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6800/115935 conn(0x7fcfa006e800 0x7fcfa0070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 20 client.4751.objecter put_session s=0x7fcfa006e150 osd=0 1 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 20 client.4751.objecter shutdown clearing up homeless session... 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 10 client.4751.objecter successfully canceled tick 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6816/115635 conn(0x7fcfa0068890 msgr2=0x7fcfa006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/3056238039 >> v2:172.21.15.115:6816/115635 conn(0x7fcfa0068890 0x7fcfa006ad40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80099b0 tx=0x7fcfa8016000 comp rx=0 tx=0).stop 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 10 monclient: shutdown 2023-09-30T01:08:43.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 msgr2=0x7fcfbc0b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 --2- 172.21.15.115:0/3056238039 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcfbc0a5640 0x7fcfbc0b3030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c5d0 tx=0x7fcfb0007f60 comp rx=0 tx=0).stop 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc10cd700 1 -- 172.21.15.115:0/3056238039 reap_dead start 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 shutdown_connections 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 >> 172.21.15.115:0/3056238039 conn(0x7fcfbc068e40 msgr2=0x7fcfbc070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 shutdown_connections 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 -- 172.21.15.115:0/3056238039 wait complete. 2023-09-30T01:08:43.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.394+0000 7fcfc2b30700 1 librados: shutdown 2023-09-30T01:08:43.413 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T01:08:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.530+0000 7f05a9153700 10 monclient: get_monmap_and_config 2023-09-30T01:08:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.530+0000 7f05a9153700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.530+0000 7f05a9153700 10 monclient: monmap: 2023-09-30T01:08:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:43.531522+0000 2023-09-30T01:08:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:43.531522+0000 2023-09-30T01:08:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 Processor -- start 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 -- start start 2023-09-30T01:08:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient: init 2023-09-30T01:08:43.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05a40ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f05a40aecc0 0x7f05a40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-a con 0x7f05a40aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-b con 0x7f05a41415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40a7040 con 0x7f05a40ae8f0 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a40a57f0 con 0x7f05a40aecc0 2023-09-30T01:08:43.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40a5480 con 0x7f05a41415c0 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a9153700 10 monclient(hunting): authenticate will time out at 1384.981748s 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f05a40aecc0 0x7f05a40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36076/0 (socket says 172.21.15.115:36076) 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f05a40aecc0 0x7f05a40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56802/0 (socket says 172.21.15.115:56802) 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 1 -- 172.21.15.115:0/634662105 learned_addr learned my addr 172.21.15.115:0/634662105 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 10 monclient(hunting): get_auth_request con 0x7f05a41415c0 auth_method 0 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a2d9d700 1 --1- 172.21.15.115:0/634662105 >> v1:172.21.15.115:6789/0 conn(0x7f05a40ae8f0 0x7f05a40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52786/0 (socket says 172.21.15.115:52786) 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 10 monclient(hunting): get_auth_request con 0x7f05a40aecc0 auth_method 0 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.531+0000 7f05a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient(hunting): handle_auth_done global_id 5034 payload 274 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 -- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 msgr2=0x7f05a40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 -- 172.21.15.115:0/634662105 >> v1:172.21.15.115:6789/0 conn(0x7f05a40ae8f0 legacy=0x7f05a40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 monclient: found mon.noname-a 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 1 -- 172.21.15.115:0/634662105 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314350240 0 0) 0x7f05a40a7040 con 0x7f05a40ae8f0 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 -- 172.21.15.115:0/634662105 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4141a40 con 0x7f05a40aecc0 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:43.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a359e700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a259c700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40a5d10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f05980098e0 tx=0x7f05980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75f32d7588c1424 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 1 -- 172.21.15.115:0/634662105 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059800b640 con 0x7f05a40aecc0 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: dump: 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.548 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 5 monclient: authenticate success, global_id 5034 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 1 -- 172.21.15.115:0/634662105 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059800b7a0 con 0x7f05a40aecc0 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 1 -- 172.21.15.115:0/634662105 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059800bb40 con 0x7f05a40aecc0 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:43.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 4 set_mon_vals no callback set 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a1d9b700 10 monclient: dump: 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.550 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 10 monclient: shutdown 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 1 -- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 msgr2=0x7f05a40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.532+0000 7f05a9153700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40a5d10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f05980098e0 tx=0x7f05980091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 -- 172.21.15.115:0/634662105 shutdown_connections 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a41415c0 0x7f05a40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 --2- 172.21.15.115:0/634662105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40a5d10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 -- 172.21.15.115:0/634662105 >> 172.21.15.115:0/634662105 conn(0x7f05a406a460 msgr2=0x7f05a40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 -- 172.21.15.115:0/634662105 shutdown_connections 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 -- 172.21.15.115:0/634662105 wait complete. 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 10 monclient: monmap: 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:43.552 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.533+0000 7f05a9153700 1 librados: starting msgr at 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 librados: starting objecter 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 Processor -- start 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 -- start start 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 librados: setting wanted keys 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 librados: calling monclient init 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient: init 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-c con 0x7f05a40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-a con 0x7f05a40aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a41415c0 0x7f05a40aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): picked mon.noname-b con 0x7f05a41415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40a9d70 con 0x7f05a40aecc0 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40ac2d0 con 0x7f05a41415c0 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40ab970 con 0x7f05a40ae8f0 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a9153700 10 monclient(hunting): authenticate will time out at 1384.984740s 2023-09-30T01:08:43.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36084/0 (socket says 172.21.15.115:36084) 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 1 -- 172.21.15.115:0/1144727144 learned_addr learned my addr 172.21.15.115:0/1144727144 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): get_auth_request con 0x7f05a40ae8f0 auth_method 0 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): get_auth_request con 0x7f05a40aecc0 auth_method 0 2023-09-30T01:08:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.534+0000 7f05a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a359e700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a41415c0 0x7f05a40aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a359e700 10 monclient(hunting): get_auth_request con 0x7f05a41415c0 auth_method 0 2023-09-30T01:08:43.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 10 monclient(hunting): handle_auth_done global_id 5040 payload 824 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 -- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 msgr2=0x7f05a413e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 -- 172.21.15.115:0/1144727144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a41415c0 msgr2=0x7f05a40aa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a41415c0 0x7f05a40aa200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 monclient: found mon.noname-a 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 -- 172.21.15.115:0/1144727144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c009ae0 con 0x7f05a40aecc0 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a2d9d700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a259c700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40abd90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0598016000 tx=0x7f0598017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 client.?.objecter ms_handle_connect 0x7f05a40aecc0 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598000d10 con 0x7f05a40aecc0 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: dump: 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.558 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059800b910 con 0x7f05a40aecc0 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 5 monclient: authenticate success, global_id 5040 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 10 monclient: _renew_subs 2023-09-30T01:08:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0598009420 con 0x7f05a40aecc0 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980266d0 con 0x7f05a40aecc0 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f058affd700 10 monclient: dump: 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.560 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.561 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.561 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.561 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.561 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a159a700 4 set_mon_vals no callback set 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 10 client.5040.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 10 monclient: _renew_subs 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a40a9320 con 0x7f05a40aecc0 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.535+0000 7f05a9153700 1 librados: init done 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f05a9153700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f05a9153700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f05a9153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a40a9ee0 con 0x7f05a40aecc0 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0598023030 con 0x7f05a40aecc0 2023-09-30T01:08:43.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.536+0000 7f058affd700 1 --2- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 0x7f059006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0598085560 con 0x7f05a40aecc0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 10 client.5040.objecter ms_dispatch 0x7f05a4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 3 client.5040.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 3 client.5040.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f05a2d9d700 1 --2- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 0x7f059006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f05a2d9d700 10 monclient: get_auth_request con 0x7f0590068890 auth_method 0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 20 client.5040.objecter dump_active .. 0 homeless 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f058affd700 10 client.5040.objecter ms_handle_connect 0x7f0590068890 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.537+0000 7f05a2d9d700 1 --2- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 0x7f059006ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0594007fd0 tx=0x7f0594007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.541+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0598052e20 con 0x7f05a40aecc0 2023-09-30T01:08:43.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.541+0000 7f058affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:43.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.541+0000 7f058affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:43.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.722+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f05a4072960 con 0x7f0590068890 2023-09-30T01:08:43.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.722+0000 7f058affd700 1 -- 172.21.15.115:0/1144727144 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f05a4072960 con 0x7f0590068890 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 10 librados: watch_flush enter 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 10 librados: watch_flush exit 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 20 client.5040.objecter shutdown clearing up homeless session... 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 10 client.5040.objecter successfully canceled tick 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 msgr2=0x7f059006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 0x7f059006ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0594007fd0 tx=0x7f0594007e10 comp rx=0 tx=0).stop 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 10 monclient: shutdown 2023-09-30T01:08:43.725 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 msgr2=0x7f05a40abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40abd90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0598016000 tx=0x7f0598017000 comp rx=0 tx=0).stop 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 shutdown_connections 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> v2:172.21.15.115:6816/115635 conn(0x7f0590068890 0x7f059006ad40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a41415c0 0x7f05a40aa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f05a40ae8f0 0x7f05a413e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 --2- 172.21.15.115:0/1144727144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f05a40aecc0 0x7f05a40abd90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 >> 172.21.15.115:0/1144727144 conn(0x7f05a406a460 msgr2=0x7f05a4144920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 shutdown_connections 2023-09-30T01:08:43.726 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 -- 172.21.15.115:0/1144727144 wait complete. 2023-09-30T01:08:43.727 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.724+0000 7f05a9153700 1 librados: shutdown 2023-09-30T01:08:43.727 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T01:08:43.736 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T01:08:43.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:43.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.856+0000 7f345b0f3700 10 monclient: get_monmap_and_config 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.856+0000 7f345b0f3700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.856+0000 7f345b0f3700 10 monclient: monmap: 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:43.857628+0000 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:43.857628+0000 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:43.857 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:43.858 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:43.858 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:43.858 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:43.858 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 1 Processor -- start 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 1 -- start start 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 10 monclient: init 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T01:08:43.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.857+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34540a5640 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f345413e040 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3454146b30 0x7f3454148f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-c con 0x7f3454146b30 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540a8220 con 0x7f345413e040 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540a6dc0 con 0x7f34540a5640 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34540a7d20 con 0x7f3454146b30 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 10 monclient(hunting): authenticate will time out at 1385.308086s 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3453fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f345413e040 0x7f345413e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52792/0 (socket says 172.21.15.115:52792) 2023-09-30T01:08:43.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3453fff700 1 -- 172.21.15.115:0/3296009172 learned_addr learned my addr 172.21.15.115:0/3296009172 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926010459 0 0) 0x7f34540a8220 con 0x7f345413e040 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 1 --2- 172.21.15.115:0/3296009172 >> v2:172.21.15.157:3300/0 conn(0x7f3454146b30 0x7f3454148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): my global_id is 5043 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3444003570 con 0x7f345413e040 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): get_auth_request con 0x7f3454146b30 auth_method 0 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3427641922 0 0) 0x7f3444003570 con 0x7f345413e040 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 >> v2:172.21.15.157:3300/0 conn(0x7f3454146b30 msgr2=0x7f3454148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 --2- 172.21.15.115:0/3296009172 >> v2:172.21.15.157:3300/0 conn(0x7f3454146b30 0x7f3454148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 >> v1:172.21.15.157:6789/0 conn(0x7f34540a5640 legacy=0x7f34540a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540a5a10 con 0x7f345413e040 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 5 monclient: authenticate success, global_id 5043 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f3440001470 con 0x7f345413e040 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: dump: 2023-09-30T01:08:43.874 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.875 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 >> v1:172.21.15.115:6789/0 conn(0x7f345413e040 legacy=0x7f345413e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 0x7f3444003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): picked mon.a con 0x7f3444003a60 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3444004810 0x7f3444004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): picked mon.b con 0x7f3444004810 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34440051c0 0x7f344400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): picked mon.c con 0x7f34440051c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540a7d20 con 0x7f3444003a60 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344400c160 con 0x7f3444004810 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344400c320 con 0x7f34440051c0 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f34537fe700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:08:43.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f345b0f3700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3458e8f700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 0x7f3444003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3458e8f700 10 monclient(hunting): get_auth_request con 0x7f3444003a60 auth_method 0 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3458e8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3458e8f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3458e8f700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3459690700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34440051c0 0x7f344400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.858+0000 7f3453fff700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3444004810 0x7f3444004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:08:43.878 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): get_auth_request con 0x7f3444004810 auth_method 0 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): get_auth_request con 0x7f34440051c0 auth_method 0 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:08:43.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient(hunting): handle_auth_done global_id 5043 payload 293 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 -- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34440051c0 msgr2=0x7f344400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f34440051c0 0x7f344400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 -- 172.21.15.115:0/3296009172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3444004810 msgr2=0x7f3444004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3444004810 0x7f3444004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 monclient: found mon.a 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3459690700 1 -- 172.21.15.115:0/3296009172 reap_dead start 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 -- 172.21.15.115:0/3296009172 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444003570 con 0x7f3444003a60 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f3458e8f700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 0x7f3444003e70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f34480018b0 tx=0x7f3448007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16535a731b94e23 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448012420 con 0x7f3444003a60 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: dump: 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.881 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480125a0 con 0x7f3444003a60 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 4 set_mon_vals no callback set 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 1 -- 172.21.15.115:0/3296009172 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448012960 con 0x7f3444003a60 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f34537fe700 10 monclient: dump: 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 10 monclient: shutdown 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 1 -- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 msgr2=0x7f3444003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.859+0000 7f345b0f3700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 0x7f3444003e70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f34480018b0 tx=0x7f3448007d30 comp rx=0 tx=0).stop 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 -- 172.21.15.115:0/3296009172 shutdown_connections 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 --2- 172.21.15.115:0/3296009172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3444003a60 0x7f3444003e70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 -- 172.21.15.115:0/3296009172 >> 172.21.15.115:0/3296009172 conn(0x7f3454068e40 msgr2=0x7f34540630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 -- 172.21.15.115:0/3296009172 shutdown_connections 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 -- 172.21.15.115:0/3296009172 wait complete. 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 10 monclient: build_initial_monmap 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 10 monclient: monmap: 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 librados: starting msgr at 2023-09-30T01:08:43.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 librados: starting objecter 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.860+0000 7f345b0f3700 1 Processor -- start 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 -- start start 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 librados: setting wanted keys 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 librados: calling monclient init 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient: init 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 0x7f34540b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-c con 0x7f3444003a60 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34540b36e0 0x7f34540ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-a con 0x7f34540b36e0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f345413fa00 0x7f34540ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): picked mon.noname-b con 0x7f345413fa00 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540b3ef0 con 0x7f34540b36e0 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540b3bc0 con 0x7f345413fa00 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3454141de0 con 0x7f3444003a60 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f345b0f3700 10 monclient(hunting): authenticate will time out at 1385.311684s 2023-09-30T01:08:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 0x7f34540b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34540b36e0 0x7f34540ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 0x7f34540b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36098/0 (socket says 172.21.15.115:36098) 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 1 -- 172.21.15.115:0/1695381211 learned_addr learned my addr 172.21.15.115:0/1695381211 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): get_auth_request con 0x7f3444003a60 auth_method 0 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): get_auth_request con 0x7f34540b36e0 auth_method 0 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3458e8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3453fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3459690700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f345413fa00 0x7f34540ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3459690700 10 monclient(hunting): get_auth_request con 0x7f345413fa00 auth_method 0 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3459690700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3459690700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:43.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.861+0000 7f3459690700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 10 monclient(hunting): handle_auth_done global_id 5046 payload 824 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 -- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 msgr2=0x7f34540b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 0x7f34540b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 -- 172.21.15.115:0/1695381211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f345413fa00 msgr2=0x7f34540ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f345413fa00 0x7f34540ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 10 monclient: _finish_hunting 0 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 monclient: found mon.noname-a 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 -- 172.21.15.115:0/1695381211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540b2aa0 con 0x7f34540b36e0 2023-09-30T01:08:43.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3458e8f700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f3444003a60 0x7f34540b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3453fff700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34540b36e0 0x7f34540ae5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f345404cc00 tx=0x7f3440001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 client.?.objecter ms_handle_connect 0x7f34540b36e0 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3440018030 con 0x7f34540b36e0 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: dump: 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.892 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344000f640 con 0x7f34540b36e0 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 5 monclient: authenticate success, global_id 5046 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 10 monclient: _renew_subs 2023-09-30T01:08:43.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345413fe50 con 0x7f34540b36e0 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000fa00 con 0x7f34540b36e0 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3450ff9700 10 monclient: dump: 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:43.894 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f3452ffd700 4 set_mon_vals no callback set 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 10 client.5046.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 10 monclient: _renew_subs 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34540ada80 con 0x7f34540b36e0 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.862+0000 7f345b0f3700 1 librados: init done 2023-09-30T01:08:43.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 10 client.5046.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 15 client.5046.objecter _session_command_op_assign -1 1 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 20 client.5046.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 10 client.5046.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f34540b2c30 con 0x7f34540b36e0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f344001b070 con 0x7f34540b36e0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.863+0000 7f3450ff9700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6816/115635 conn(0x7f3438068890 0x7f343806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3458e8f700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6816/115635 conn(0x7f3438068890 0x7f343806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3440019070 con 0x7f34540b36e0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3458e8f700 10 monclient: get_auth_request con 0x7f3438068890 auth_method 0 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 client.5046.objecter ms_dispatch 0x7f3454001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:43.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 3 client.5046.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 3 client.5046.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 client.5046.objecter checking command 1 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6808/115950 conn(0x7f343806e800 0x7f3438070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter _get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 15 client.5046.objecter _session_command_op_remove -1 1 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter _get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 15 client.5046.objecter _session_command_op_assign 1 1 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 4 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 client.5046.objecter _send_command 1 2023-09-30T01:08:43.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 --> v2:172.21.15.115:6808/115950 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3438071280 con 0x7f343806e800 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 20 client.5046.objecter dump_active .. 0 homeless 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3440088240 con 0x7f34540b36e0 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3459690700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6808/115950 conn(0x7f343806e800 0x7f3438070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3459690700 10 monclient: get_auth_request con 0x7f343806e800 auth_method 0 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 client.5046.objecter ms_handle_connect 0x7f3438068890 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3458e8f700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6816/115635 conn(0x7f3438068890 0x7f343806ad40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3448012ad0 tx=0x7f3448015000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3450ff9700 10 client.5046.objecter ms_handle_connect 0x7f343806e800 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.864+0000 7f3459690700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6808/115950 conn(0x7f343806e800 0x7f3438070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:43.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.865+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== osd.1 v2:172.21.15.115:6808/115950 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3438071280 con 0x7f343806e800 2023-09-30T01:08:43.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.865+0000 7f3450ff9700 10 client.5046.objecter ms_dispatch 0x7f3454001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:43.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.865+0000 7f3450ff9700 10 client.5046.objecter _finish_command 1 = system:0 2023-09-30T01:08:43.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.865+0000 7f3450ff9700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 4 2023-09-30T01:08:43.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.865+0000 7f3450ff9700 15 client.5046.objecter _session_command_op_remove 1 1 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 10 client.5046.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 15 client.5046.objecter _session_command_op_assign -1 2 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 20 client.5046.objecter _get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 20 client.5046.objecter _get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 15 client.5046.objecter _session_command_op_remove -1 2 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 20 client.5046.objecter get_session s=0x7f343806e150 osd=1 3 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 15 client.5046.objecter _session_command_op_assign 1 2 2023-09-30T01:08:43.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 4 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 10 client.5046.objecter _send_command 2 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.878+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 --> v2:172.21.15.115:6808/115950 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f345413f740 con 0x7f343806e800 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3450ff9700 1 -- 172.21.15.115:0/1695381211 <== osd.1 v2:172.21.15.115:6808/115950 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f345413f740 con 0x7f343806e800 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3450ff9700 10 client.5046.objecter ms_dispatch 0x7f3454001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3450ff9700 10 client.5046.objecter _finish_command 2 = system:0 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3450ff9700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 4 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3450ff9700 15 client.5046.objecter _session_command_op_remove 1 2 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:43.901 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:43.902 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:43.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 10 librados: watch_flush enter 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 10 librados: watch_flush exit 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 10 client.5046.objecter close_session for osd.1 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6808/115950 conn(0x7f343806e800 msgr2=0x7f3438070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6808/115950 conn(0x7f343806e800 0x7f3438070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 20 client.5046.objecter put_session s=0x7f343806e150 osd=1 1 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 20 client.5046.objecter shutdown clearing up homeless session... 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 10 client.5046.objecter successfully canceled tick 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6816/115635 conn(0x7f3438068890 msgr2=0x7f343806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 --2- 172.21.15.115:0/1695381211 >> v2:172.21.15.115:6816/115635 conn(0x7f3438068890 0x7f343806ad40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3448012ad0 tx=0x7f3448015000 comp rx=0 tx=0).stop 2023-09-30T01:08:43.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 10 monclient: shutdown 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34540b36e0 msgr2=0x7f34540ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 --2- 172.21.15.115:0/1695381211 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f34540b36e0 0x7f34540ae5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f345404cc00 tx=0x7f3440001e50 comp rx=0 tx=0).stop 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f3459690700 1 -- 172.21.15.115:0/1695381211 reap_dead start 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 shutdown_connections 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 >> 172.21.15.115:0/1695381211 conn(0x7f3454068e40 msgr2=0x7f345413a030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 shutdown_connections 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 -- 172.21.15.115:0/1695381211 wait complete. 2023-09-30T01:08:43.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:43.879+0000 7f345b0f3700 1 librados: shutdown 2023-09-30T01:08:43.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T01:08:44.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.022+0000 7f983d227700 10 monclient: get_monmap_and_config 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.022+0000 7f983d227700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.022+0000 7f983d227700 10 monclient: monmap: 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:44.023678+0000 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:44.023678+0000 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:44.024 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 Processor -- start 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 -- start start 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient: init 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f98380a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): picked mon.noname-b con 0x7f98380ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98380aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98381415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380a7040 con 0x7f98380aecc0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380a57f0 con 0x7f98381415c0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380a5480 con 0x7f98380ae8f0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.023+0000 7f983d227700 10 monclient(hunting): authenticate will time out at 1385.473989s 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f98380a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983659c700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f98380aecc0 0x7f98380b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52794/0 (socket says 172.21.15.115:52794) 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983659c700 1 -- 172.21.15.115:0/2188598502 learned_addr learned my addr 172.21.15.115:0/2188598502 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): get_auth_request con 0x7f98380ae8f0 auth_method 0 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 1 -- 172.21.15.115:0/2188598502 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296977745 0 0) 0x7f98380a7040 con 0x7f98380aecc0 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient(hunting): my global_id is 5049 2023-09-30T01:08:44.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 1 -- 172.21.15.115:0/2188598502 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003570 con 0x7f98380aecc0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 -- 172.21.15.115:0/2188598502 >> v1:172.21.15.157:6789/0 conn(0x7f98381415c0 legacy=0x7f98380adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 -- 172.21.15.115:0/2188598502 >> v1:172.21.15.115:6789/0 conn(0x7f98380aecc0 legacy=0x7f98380b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 monclient: found mon.noname-b 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 -- 172.21.15.115:0/2188598502 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380a55c0 con 0x7f98380ae8f0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9836d9d700 1 --2- 172.21.15.115:0/2188598502 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f98380a5d10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9828009910 tx=0x7f98280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961d1758c75db040 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 1 -- 172.21.15.115:0/2188598502 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800b680 con 0x7f98380ae8f0 2023-09-30T01:08:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: dump: 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983d227700 5 monclient: authenticate success, global_id 4763 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983d227700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 1 -- 172.21.15.115:0/2188598502 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982800b7e0 con 0x7f98380ae8f0 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983d227700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f983d227700 4 set_mon_vals no callback set 2023-09-30T01:08:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 1 -- 172.21.15.115:0/2188598502 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800bba0 con 0x7f98380ae8f0 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.024+0000 7f9835d9b700 10 monclient: dump: 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 10 monclient: shutdown 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 -- 172.21.15.115:0/2188598502 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 msgr2=0x7f98380a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 --2- 172.21.15.115:0/2188598502 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f98380a5d10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9828009910 tx=0x7f98280091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 -- 172.21.15.115:0/2188598502 shutdown_connections 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 --2- 172.21.15.115:0/2188598502 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f98380a5d10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 -- 172.21.15.115:0/2188598502 >> 172.21.15.115:0/2188598502 conn(0x7f983806a460 msgr2=0x7f98380630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 -- 172.21.15.115:0/2188598502 shutdown_connections 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.025+0000 7f983d227700 1 -- 172.21.15.115:0/2188598502 wait complete. 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient: monmap: 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 librados: starting msgr at 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 librados: starting objecter 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 Processor -- start 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 -- start start 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 librados: setting wanted keys 2023-09-30T01:08:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 librados: calling monclient init 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient: init 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.026+0000 7f983d227700 10 monclient(hunting): picked mon.noname-c con 0x7f98380ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): picked mon.noname-a con 0x7f98380aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 0x7f98380ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): picked mon.noname-b con 0x7f98381415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983811c820 con 0x7f98380aecc0 2023-09-30T01:08:44.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380abd90 con 0x7f98381415c0 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380ac040 con 0x7f98380ae8f0 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient(hunting): authenticate will time out at 1385.477055s 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36114/0 (socket says 172.21.15.115:36114) 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 1 -- 172.21.15.115:0/1558311078 learned_addr learned my addr 172.21.15.115:0/1558311078 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56848/0 (socket says 172.21.15.115:56848) 2023-09-30T01:08:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): get_auth_request con 0x7f98380ae8f0 auth_method 0 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): get_auth_request con 0x7f98380aecc0 auth_method 0 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 0x7f98380ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 10 monclient(hunting): get_auth_request con 0x7f98381415c0 auth_method 0 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient(hunting): handle_auth_done global_id 5052 payload 824 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 -- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 msgr2=0x7f983814dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 -- 172.21.15.115:0/1558311078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 msgr2=0x7f98380ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 0x7f98380ac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 monclient: found mon.noname-a 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 -- 172.21.15.115:0/1558311078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828009420 con 0x7f98380aecc0 2023-09-30T01:08:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f9836d9d700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983759e700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 0x7f98380ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983659c700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c5b0 tx=0x7f982c007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 client.?.objecter ms_handle_connect 0x7f98380aecc0 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c005c00 con 0x7f98380aecc0 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: dump: 2023-09-30T01:08:44.048 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 5 monclient: authenticate success, global_id 5052 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c005d80 con 0x7f98380aecc0 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient: _renew_subs 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98380a8df0 con 0x7f98380aecc0 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c007320 con 0x7f98380aecc0 2023-09-30T01:08:44.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f981effd700 10 monclient: dump: 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.051 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983559a700 4 set_mon_vals no callback set 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 client.5052.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient: _renew_subs 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983814eb50 con 0x7f98380aecc0 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.027+0000 7f983d227700 1 librados: init done 2023-09-30T01:08:44.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.028+0000 7f983d227700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.028+0000 7f983d227700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.028+0000 7f983d227700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.028+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9838072960 con 0x7f98380aecc0 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f982c01b3b0 con 0x7f98380aecc0 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 1 --2- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 0x7f982406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f982c0833a0 con 0x7f98380aecc0 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 10 client.5052.objecter ms_dispatch 0x7f9838001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 3 client.5052.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 3 client.5052.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:44.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f9836d9d700 1 --2- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 0x7f982406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f9836d9d700 10 monclient: get_auth_request con 0x7f9824068890 auth_method 0 2023-09-30T01:08:44.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 20 client.5052.objecter dump_active .. 0 homeless 2023-09-30T01:08:44.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f981effd700 10 client.5052.objecter ms_handle_connect 0x7f9824068890 2023-09-30T01:08:44.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.029+0000 7f9836d9d700 1 --2- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 0x7f982406ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f98280091e0 tx=0x7f9828017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.033+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f982c050c60 con 0x7f98380aecc0 2023-09-30T01:08:44.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.033+0000 7f981effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.033+0000 7f981effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:44.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.217+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9838141a40 con 0x7f9824068890 2023-09-30T01:08:44.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.217+0000 7f981effd700 1 -- 172.21.15.115:0/1558311078 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9838141a40 con 0x7f9824068890 2023-09-30T01:08:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 10 librados: watch_flush enter 2023-09-30T01:08:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 10 librados: watch_flush exit 2023-09-30T01:08:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 20 client.5052.objecter shutdown clearing up homeless session... 2023-09-30T01:08:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 10 client.5052.objecter successfully canceled tick 2023-09-30T01:08:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 msgr2=0x7f982406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 0x7f982406ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f98280091e0 tx=0x7f9828017000 comp rx=0 tx=0).stop 2023-09-30T01:08:44.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 10 monclient: shutdown 2023-09-30T01:08:44.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 msgr2=0x7f98380ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c5b0 tx=0x7f982c007d50 comp rx=0 tx=0).stop 2023-09-30T01:08:44.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 shutdown_connections 2023-09-30T01:08:44.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> v2:172.21.15.115:6816/115635 conn(0x7f9824068890 0x7f982406ad40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98381415c0 0x7f98380ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f98380ae8f0 0x7f983814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 --2- 172.21.15.115:0/1558311078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98380aecc0 0x7f98380ab850 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 >> 172.21.15.115:0/1558311078 conn(0x7f983806a460 msgr2=0x7f983813b020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 shutdown_connections 2023-09-30T01:08:44.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 -- 172.21.15.115:0/1558311078 wait complete. 2023-09-30T01:08:44.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.219+0000 7f983d227700 1 librados: shutdown 2023-09-30T01:08:44.224 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T01:08:44.231 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T01:08:44.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:44.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.351+0000 7f3773192700 10 monclient: get_monmap_and_config 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.351+0000 7f3773192700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.351+0000 7f3773192700 10 monclient: monmap: 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:44.352814+0000 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:44.352814+0000 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:44.352 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:44.353 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:44.353 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:44.353 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:44.353 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.352+0000 7f3773192700 1 Processor -- start 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.352+0000 7f3773192700 1 -- start start 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.352+0000 7f3773192700 10 monclient: init 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): picked mon.noname-b con 0x7f376c0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): picked mon.noname-a con 0x7f376c13e040 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:44.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f376c141590 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c0a8220 con 0x7f376c141590 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c0a6dc0 con 0x7f376c13e040 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c0a7d20 con 0x7f376c0a5640 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3773192700 10 monclient(hunting): authenticate will time out at 1385.803336s 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56856/0 (socket says 172.21.15.115:56856) 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 1 -- 172.21.15.115:0/373213594 learned_addr learned my addr 172.21.15.115:0/373213594 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): get_auth_request con 0x7f376c13e040 auth_method 0 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36116/0 (socket says 172.21.15.115:36116) 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): get_auth_request con 0x7f376c0a5640 auth_method 0 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 -- 172.21.15.115:0/373213594 >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 msgr2=0x7f376c13e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 --2- 172.21.15.115:0/373213594 >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 -- 172.21.15.115:0/373213594 >> v1:172.21.15.157:6789/0 conn(0x7f376c141590 legacy=0x7f376c1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 monclient: found mon.noname-b 2023-09-30T01:08:44.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 -- 172.21.15.115:0/373213594 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0a5a10 con 0x7f376c0a5640 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376bfff700 1 --2- 172.21.15.115:0/373213594 >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f3770f2e700 1 --2- 172.21.15.115:0/373213594 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f37600098e0 tx=0x7f37600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a83e328e75f6ac4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376b7fe700 1 -- 172.21.15.115:0/373213594 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760011630 con 0x7f376c0a5640 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.353+0000 7f376b7fe700 10 monclient: dump: 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.367 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 5 monclient: authenticate success, global_id 4769 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 1 -- 172.21.15.115:0/373213594 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3760011790 con 0x7f376c0a5640 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 1 -- 172.21.15.115:0/373213594 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760011b30 con 0x7f376c0a5640 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 4 set_mon_vals no callback set 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f376b7fe700 10 monclient: dump: 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.369 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 10 monclient: shutdown 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 -- 172.21.15.115:0/373213594 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 msgr2=0x7f376c0a74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 --2- 172.21.15.115:0/373213594 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f37600098e0 tx=0x7f37600091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 -- 172.21.15.115:0/373213594 shutdown_connections 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 --2- 172.21.15.115:0/373213594 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c0a74c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 --2- 172.21.15.115:0/373213594 >> v2:172.21.15.115:3300/0 conn(0x7f376c13e040 0x7f376c13e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 -- 172.21.15.115:0/373213594 >> 172.21.15.115:0/373213594 conn(0x7f376c068e40 msgr2=0x7f376c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 -- 172.21.15.115:0/373213594 shutdown_connections 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.354+0000 7f3773192700 1 -- 172.21.15.115:0/373213594 wait complete. 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 10 monclient: monmap: 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.371 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 librados: starting msgr at 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 librados: starting objecter 2023-09-30T01:08:44.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 Processor -- start 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 -- start start 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 librados: setting wanted keys 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 1 librados: calling monclient init 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.355+0000 7f3773192700 10 monclient: init 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c1511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): picked mon.noname-c con 0x7f376c0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f376c13e040 0x7f376c1516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): picked mon.noname-b con 0x7f376c13e040 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f376c141590 0x7f376c13fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): picked mon.noname-a con 0x7f376c141590 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c13f9a0 con 0x7f376c141590 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c0a85d0 con 0x7f376c13e040 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c151c20 con 0x7f376c0a5640 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 10 monclient(hunting): authenticate will time out at 1385.806306s 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c1511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c1511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36128/0 (socket says 172.21.15.115:36128) 2023-09-30T01:08:44.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 -- 172.21.15.115:0/3818802027 learned_addr learned my addr 172.21.15.115:0/3818802027 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): get_auth_request con 0x7f376c0a5640 auth_method 0 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f376c141590 0x7f376c13fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): get_auth_request con 0x7f376c141590 auth_method 0 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f376bfff700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f376c13e040 0x7f376c1516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f376bfff700 10 monclient(hunting): get_auth_request con 0x7f376c13e040 auth_method 0 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f376bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f376bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f376bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient(hunting): handle_auth_done global_id 4772 payload 824 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 -- 172.21.15.115:0/3818802027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f376c13e040 msgr2=0x7f376c1516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f376c13e040 0x7f376c1516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 -- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f376c141590 msgr2=0x7f376c13fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f376c141590 0x7f376c13fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 monclient: found mon.noname-c 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 -- 172.21.15.115:0/3818802027 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094f0 con 0x7f376c0a5640 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f377172f700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f376c141590 0x7f376c13fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3770f2e700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c1511a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37600093f0 tx=0x7f3760016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 client.?.objecter ms_handle_connect 0x7f376c0a5640 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376001bdf0 con 0x7f376c0a5640 2023-09-30T01:08:44.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 monclient: dump: 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.378 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376001c450 con 0x7f376c0a5640 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3773192700 5 monclient: authenticate success, global_id 4772 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.356+0000 7f3768ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376001c810 con 0x7f376c0a5640 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 monclient: _renew_subs 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3760009420 con 0x7f376c0a5640 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3768ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f376affd700 4 set_mon_vals no callback set 2023-09-30T01:08:44.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3768ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3768ff9700 10 monclient: dump: 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 client.4772.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 monclient: _renew_subs 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c1405b0 con 0x7f376c0a5640 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 1 librados: init done 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 client.4772.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 15 client.4772.objecter _session_command_op_assign -1 1 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 20 client.4772.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 client.4772.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.357+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f376c072960 con 0x7f376c0a5640 2023-09-30T01:08:44.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f376001cc00 con 0x7f376c0a5640 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6816/115635 conn(0x7f3750068890 0x7f375006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3760085780 con 0x7f376c0a5640 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 10 client.4772.objecter ms_dispatch 0x7f376c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 3 client.4772.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 3 client.4772.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f376bfff700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6816/115635 conn(0x7f3750068890 0x7f375006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f376bfff700 10 monclient: get_auth_request con 0x7f3750068890 auth_method 0 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 10 client.4772.objecter checking command 1 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6804/115951 conn(0x7f375006e800 0x7f3750070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter _get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 15 client.4772.objecter _session_command_op_remove -1 1 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter _get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 15 client.4772.objecter _session_command_op_assign 2 1 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 4 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 10 client.4772.objecter _send_command 1 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 --> v2:172.21.15.115:6804/115951 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3750071280 con 0x7f375006e800 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 20 client.4772.objecter dump_active .. 0 homeless 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3760087d60 con 0x7f376c0a5640 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f3768ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f377172f700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6804/115951 conn(0x7f375006e800 0x7f3750070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.358+0000 7f377172f700 10 monclient: get_auth_request con 0x7f375006e800 auth_method 0 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 10 client.4772.objecter ms_handle_connect 0x7f3750068890 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f376bfff700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6816/115635 conn(0x7f3750068890 0x7f375006ad40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f376c0b0b50 tx=0x7f3758008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f377172f700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6804/115951 conn(0x7f375006e800 0x7f3750070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 10 client.4772.objecter ms_handle_connect 0x7f375006e800 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== osd.2 v2:172.21.15.115:6804/115951 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3750071280 con 0x7f375006e800 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 10 client.4772.objecter ms_dispatch 0x7f376c001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 10 client.4772.objecter _finish_command 1 = system:0 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 4 2023-09-30T01:08:44.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.359+0000 7f3768ff9700 15 client.4772.objecter _session_command_op_remove 2 1 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 client.4772.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 15 client.4772.objecter _session_command_op_assign -1 2 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter _get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter _get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 15 client.4772.objecter _session_command_op_remove -1 2 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter get_session s=0x7f375006e150 osd=2 3 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 15 client.4772.objecter _session_command_op_assign 2 2 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 4 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 client.4772.objecter _send_command 2 2023-09-30T01:08:44.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 --> v2:172.21.15.115:6804/115951 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f376c0a7a40 con 0x7f375006e800 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3768ff9700 1 -- 172.21.15.115:0/3818802027 <== osd.2 v2:172.21.15.115:6804/115951 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f376c0a7a40 con 0x7f375006e800 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3768ff9700 10 client.4772.objecter ms_dispatch 0x7f376c001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3768ff9700 10 client.4772.objecter _finish_command 2 = system:0 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3768ff9700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 4 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3768ff9700 15 client.4772.objecter _session_command_op_remove 2 2 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:44.387 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:44.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 librados: watch_flush enter 2023-09-30T01:08:44.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 librados: watch_flush exit 2023-09-30T01:08:44.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 client.4772.objecter close_session for osd.2 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6804/115951 conn(0x7f375006e800 msgr2=0x7f3750070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6804/115951 conn(0x7f375006e800 0x7f3750070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter put_session s=0x7f375006e150 osd=2 1 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 client.4772.objecter shutdown clearing up homeless session... 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 client.4772.objecter successfully canceled tick 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6816/115635 conn(0x7f3750068890 msgr2=0x7f375006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 --2- 172.21.15.115:0/3818802027 >> v2:172.21.15.115:6816/115635 conn(0x7f3750068890 0x7f375006ad40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f376c0b0b50 tx=0x7f3758008040 comp rx=0 tx=0).stop 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 10 monclient: shutdown 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 msgr2=0x7f376c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 --2- 172.21.15.115:0/3818802027 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f376c0a5640 0x7f376c1511a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37600093f0 tx=0x7f3760016000 comp rx=0 tx=0).stop 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f377172f700 1 -- 172.21.15.115:0/3818802027 reap_dead start 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 shutdown_connections 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 >> 172.21.15.115:0/3818802027 conn(0x7f376c068e40 msgr2=0x7f376c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 shutdown_connections 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.373+0000 7f3773192700 1 -- 172.21.15.115:0/3818802027 wait complete. 2023-09-30T01:08:44.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.374+0000 7f3773192700 1 librados: shutdown 2023-09-30T01:08:44.391 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 10 monclient: get_monmap_and_config 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 10 monclient: monmap: 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.509 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:44.510140+0000 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:44.510140+0000 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:44.510 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 1 Processor -- start 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 1 -- start start 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.509+0000 7fb069acf700 10 monclient: init 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 0x7fb0640a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-c con 0x7fb0640ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb0640aecc0 0x7fb0640b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.520 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-a con 0x7fb0640aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0641415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0640a7040 con 0x7fb0641415c0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640b19b0 con 0x7fb0640aecc0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640a57f0 con 0x7fb0640ae8f0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb069acf700 10 monclient(hunting): authenticate will time out at 1385.960548s 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb0640aecc0 0x7fb0640b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fb0640aecc0 0x7fb0640b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56882/0 (socket says 172.21.15.115:56882) 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 -- 172.21.15.115:0/3122709655 learned_addr learned my addr 172.21.15.115:0/3122709655 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): get_auth_request con 0x7fb0640aecc0 auth_method 0 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient(hunting): handle_auth_done global_id 5061 payload 274 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb0637fe700 1 --2- 172.21.15.115:0/3122709655 >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 0x7fb0640a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 -- 172.21.15.115:0/3122709655 >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 msgr2=0x7fb0640a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 --2- 172.21.15.115:0/3122709655 >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 0x7fb0640a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 -- 172.21.15.115:0/3122709655 >> v1:172.21.15.157:6789/0 conn(0x7fb0641415c0 legacy=0x7fb0640adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 monclient: found mon.noname-a 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb062ffd700 1 -- 172.21.15.115:0/3122709655 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0640a5480 con 0x7fb0640aecc0 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb0637fe700 10 monclient: get_auth_request con 0x7fb0640ae8f0 auth_method 0 2023-09-30T01:08:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.510+0000 7fb0637fe700 1 --2- 172.21.15.115:0/3122709655 >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 0x7fb0640a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb062ffd700 1 --2- 172.21.15.115:0/3122709655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640aecc0 0x7fb0640b1470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b59c0 tx=0x7fb054009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37744fd60c7a520a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 1 -- 172.21.15.115:0/3122709655 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400b820 con 0x7fb0640aecc0 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: dump: 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.524 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 5 monclient: authenticate success, global_id 5061 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 1 -- 172.21.15.115:0/3122709655 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05400b980 con 0x7fb0640aecc0 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 1 -- 172.21.15.115:0/3122709655 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400bd20 con 0x7fb0640aecc0 2023-09-30T01:08:44.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 4 set_mon_vals no callback set 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb0627fc700 10 monclient: dump: 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.527 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 10 monclient: shutdown 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 -- 172.21.15.115:0/3122709655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640aecc0 msgr2=0x7fb0640b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 --2- 172.21.15.115:0/3122709655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640aecc0 0x7fb0640b1470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b59c0 tx=0x7fb054009250 comp rx=0 tx=0).stop 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 -- 172.21.15.115:0/3122709655 shutdown_connections 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 --2- 172.21.15.115:0/3122709655 >> v2:172.21.15.157:3300/0 conn(0x7fb0640ae8f0 0x7fb0640a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 --2- 172.21.15.115:0/3122709655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640aecc0 0x7fb0640b1470 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 -- 172.21.15.115:0/3122709655 >> 172.21.15.115:0/3122709655 conn(0x7fb06406a460 msgr2=0x7fb0640630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 -- 172.21.15.115:0/3122709655 shutdown_connections 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.511+0000 7fb069acf700 1 -- 172.21.15.115:0/3122709655 wait complete. 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 10 monclient: monmap: 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:44.529 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 librados: starting msgr at 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 librados: starting objecter 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 Processor -- start 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 -- start start 2023-09-30T01:08:44.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 librados: setting wanted keys 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 1 librados: calling monclient init 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.512+0000 7fb069acf700 10 monclient: init 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-a con 0x7fb0640ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0640aecc0 0x7fb0641512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-b con 0x7fb0640aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb0641415c0 0x7fb0640a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): picked mon.noname-c con 0x7fb0641415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640a8ee0 con 0x7fb0640ae8f0 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640ab2d0 con 0x7fb0640aecc0 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641517f0 con 0x7fb0641415c0 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb069acf700 10 monclient(hunting): authenticate will time out at 1385.963448s 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56896/0 (socket says 172.21.15.115:56896) 2023-09-30T01:08:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 -- 172.21.15.115:0/2073169469 learned_addr learned my addr 172.21.15.115:0/2073169469 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): get_auth_request con 0x7fb0640ae8f0 auth_method 0 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb0641415c0 0x7fb0640a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): get_auth_request con 0x7fb0641415c0 auth_method 0 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0640aecc0 0x7fb0641512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): get_auth_request con 0x7fb0640aecc0 auth_method 0 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb063fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb062ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient(hunting): handle_auth_done global_id 5064 payload 824 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 -- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb0641415c0 msgr2=0x7fb0640a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb0641415c0 0x7fb0640a9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 -- 172.21.15.115:0/2073169469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0640aecc0 msgr2=0x7fb0641512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0640aecc0 0x7fb0641512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 monclient: found mon.noname-a 2023-09-30T01:08:44.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 -- 172.21.15.115:0/2073169469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540094f0 con 0x7fb0640ae8f0 2023-09-30T01:08:44.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb0637fe700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00c830 tx=0x7fb04c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb05b7fe700 10 client.?.objecter ms_handle_connect 0x7fb0640ae8f0 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.513+0000 7fb05b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c011600 con 0x7fb0640ae8f0 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: dump: 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.537 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c011760 con 0x7fb0640ae8f0 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 5 monclient: authenticate success, global_id 5064 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c011b20 con 0x7fb0640ae8f0 2023-09-30T01:08:44.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 10 monclient: _renew_subs 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb064151a10 con 0x7fb0640ae8f0 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb061ffb700 4 set_mon_vals no callback set 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb05b7fe700 10 monclient: dump: 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.539 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 10 client.5064.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 10 monclient: _renew_subs 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0640a9ab0 con 0x7fb0640ae8f0 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.514+0000 7fb069acf700 1 librados: init done 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb069acf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb069acf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb069acf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb064151c00 con 0x7fb0640ae8f0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb04c018360 con 0x7fb0640ae8f0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 1 --2- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 0x7fb04406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb04c083e30 con 0x7fb0640ae8f0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 10 client.5064.objecter ms_dispatch 0x7fb064001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 3 client.5064.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb05b7fe700 3 client.5064.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb062ffd700 1 --2- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 0x7fb04406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.515+0000 7fb062ffd700 10 monclient: get_auth_request con 0x7fb044068890 auth_method 0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.516+0000 7fb05b7fe700 20 client.5064.objecter dump_active .. 0 homeless 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.516+0000 7fb05b7fe700 10 client.5064.objecter ms_handle_connect 0x7fb044068890 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.516+0000 7fb062ffd700 1 --2- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 0x7fb04406ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b59c0 tx=0x7fb054016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.520+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb04c0516f0 con 0x7fb0640ae8f0 2023-09-30T01:08:44.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.520+0000 7fb05b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.520+0000 7fb05b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:44.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.702+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0640a9180 con 0x7fb044068890 2023-09-30T01:08:44.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.702+0000 7fb05b7fe700 1 -- 172.21.15.115:0/2073169469 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb0640a9180 con 0x7fb044068890 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 10 librados: watch_flush enter 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 10 librados: watch_flush exit 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 20 client.5064.objecter shutdown clearing up homeless session... 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 10 client.5064.objecter successfully canceled tick 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 msgr2=0x7fb04406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 0x7fb04406ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b59c0 tx=0x7fb054016000 comp rx=0 tx=0).stop 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 10 monclient: shutdown 2023-09-30T01:08:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 msgr2=0x7fb06413dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00c830 tx=0x7fb04c008040 comp rx=0 tx=0).stop 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 shutdown_connections 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> v2:172.21.15.115:6816/115635 conn(0x7fb044068890 0x7fb04406ad40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fb0641415c0 0x7fb0640a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb0640ae8f0 0x7fb06413dfd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 --2- 172.21.15.115:0/2073169469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0640aecc0 0x7fb0641512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 >> 172.21.15.115:0/2073169469 conn(0x7fb06406a460 msgr2=0x7fb064144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 shutdown_connections 2023-09-30T01:08:44.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 -- 172.21.15.115:0/2073169469 wait complete. 2023-09-30T01:08:44.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.704+0000 7fb069acf700 1 librados: shutdown 2023-09-30T01:08:44.707 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T01:08:44.716 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T01:08:44.716 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:44.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.836+0000 7f7707114700 10 monclient: get_monmap_and_config 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.836+0000 7f7707114700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.836+0000 7f7707114700 10 monclient: monmap: 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:44.837686+0000 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:44.837686+0000 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:44.837 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:44.838 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:44.838 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:44.838 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.837+0000 7f7707114700 1 Processor -- start 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.837+0000 7f7707114700 1 -- start start 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.837+0000 7f7707114700 10 monclient: init 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 0x7f77000a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): picked mon.noname-b con 0x7f77000a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f770013e040 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f770013acd0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770013eba0 con 0x7f770013e040 2023-09-30T01:08:44.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77000a8220 con 0x7f770013acd0 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77000a6dc0 con 0x7f77000a5640 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient(hunting): authenticate will time out at 1386.288149s 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 0x7f77000a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76fffff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f770013e040 0x7f770013e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52808/0 (socket says 172.21.15.115:52808) 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76fffff700 1 -- 172.21.15.115:0/1884816876 learned_addr learned my addr 172.21.15.115:0/1884816876 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): get_auth_request con 0x7f77000a5640 auth_method 0 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 1 -- 172.21.15.115:0/1884816876 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010784458 0 0) 0x7f770013eba0 con 0x7f770013e040 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient(hunting): my global_id is 5067 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 1 -- 172.21.15.115:0/1884816876 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f0003570 con 0x7f770013e040 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 -- 172.21.15.115:0/1884816876 >> v1:172.21.15.157:6789/0 conn(0x7f770013acd0 legacy=0x7f770013b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 -- 172.21.15.115:0/1884816876 >> v1:172.21.15.115:6789/0 conn(0x7f770013e040 legacy=0x7f770013e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 monclient: found mon.noname-b 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 -- 172.21.15.115:0/1884816876 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77000a7d20 con 0x7f77000a5640 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7704eb0700 1 --2- 172.21.15.115:0/1884816876 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 0x7f77000a74c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009910 tx=0x7f76f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2b90419f697d5be server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 1 -- 172.21.15.115:0/1884816876 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f400b640 con 0x7f77000a5640 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: dump: 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.851 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 5 monclient: authenticate success, global_id 4778 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 1 -- 172.21.15.115:0/1884816876 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f400b7a0 con 0x7f77000a5640 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f7707114700 4 set_mon_vals no callback set 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 1 -- 172.21.15.115:0/1884816876 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f400bb20 con 0x7f77000a5640 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.838+0000 7f76ff7fe700 10 monclient: dump: 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.853 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 10 monclient: shutdown 2023-09-30T01:08:44.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 -- 172.21.15.115:0/1884816876 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 msgr2=0x7f77000a74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 --2- 172.21.15.115:0/1884816876 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 0x7f77000a74c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009910 tx=0x7f76f40091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 -- 172.21.15.115:0/1884816876 shutdown_connections 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 --2- 172.21.15.115:0/1884816876 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f77000a5640 0x7f77000a74c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 -- 172.21.15.115:0/1884816876 >> 172.21.15.115:0/1884816876 conn(0x7f7700068e40 msgr2=0x7f77000630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 -- 172.21.15.115:0/1884816876 shutdown_connections 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.839+0000 7f7707114700 1 -- 172.21.15.115:0/1884816876 wait complete. 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 10 monclient: monmap: 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.855 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 librados: starting msgr at 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 librados: starting objecter 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 Processor -- start 2023-09-30T01:08:44.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 -- start start 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 librados: setting wanted keys 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 1 librados: calling monclient init 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.840+0000 7f7707114700 10 monclient: init 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77000a5640 0x7f7700150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): picked mon.noname-b con 0x7f77000a5640 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 0x7f7700151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): picked mon.noname-c con 0x7f770013acd0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f770013e040 0x7f77000a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): picked mon.noname-a con 0x7f770013e040 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000a94e0 con 0x7f770013e040 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000ae900 con 0x7f77000a5640 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7700151990 con 0x7f770013acd0 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7707114700 10 monclient(hunting): authenticate will time out at 1386.291579s 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 0x7f7700151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f770013e040 0x7f77000a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 0x7f7700151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36146/0 (socket says 172.21.15.115:36146) 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): get_auth_request con 0x7f770013e040 auth_method 0 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 1 -- 172.21.15.115:0/3343540665 learned_addr learned my addr 172.21.15.115:0/3343540665 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): get_auth_request con 0x7f770013acd0 auth_method 0 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77000a5640 0x7f7700150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 10 monclient(hunting): get_auth_request con 0x7f77000a5640 auth_method 0 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient(hunting): handle_auth_done global_id 5070 payload 824 2023-09-30T01:08:44.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 -- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 msgr2=0x7f7700151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 0x7f7700151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 -- 172.21.15.115:0/3343540665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77000a5640 msgr2=0x7f7700150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77000a5640 0x7f7700150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient: _finish_hunting 0 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 monclient: found mon.noname-a 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f77056b1700 1 -- 172.21.15.115:0/3343540665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f4009420 con 0x7f770013e040 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f7704eb0700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77000a5640 0x7f7700150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:08:44.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.841+0000 7f76fffff700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f770013acd0 0x7f7700151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f77056b1700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f770013e040 0x7f77000a9930 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f76f800c840 tx=0x7f76f8002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 client.?.objecter ms_handle_connect 0x7f770013e040 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8012630 con 0x7f770013e040 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: dump: 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.862 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:44.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f80127b0 con 0x7f770013e040 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 5 monclient: authenticate success, global_id 5070 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 10 monclient: _renew_subs 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77000a9e70 con 0x7f770013e040 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8012b70 con 0x7f770013e040 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76fcff9700 10 monclient: dump: 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:44.864 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f76feffd700 4 set_mon_vals no callback set 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 10 client.5070.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 10 monclient: _renew_subs 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77000aa430 con 0x7f770013e040 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.842+0000 7f7707114700 1 librados: init done 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 10 client.5070.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 15 client.5070.objecter _session_command_op_assign -1 1 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 20 client.5070.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 10 client.5070.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7700072960 con 0x7f770013e040 2023-09-30T01:08:44.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f76f801a3b0 con 0x7f770013e040 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6816/115635 conn(0x7f76e4068890 0x7f76e406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f76f8083b50 con 0x7f770013e040 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 10 client.5070.objecter ms_dispatch 0x7f7700001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f7704eb0700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6816/115635 conn(0x7f76e4068890 0x7f76e406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 3 client.5070.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.843+0000 7f76fcff9700 3 client.5070.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f7704eb0700 10 monclient: get_auth_request con 0x7f76e4068890 auth_method 0 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 10 client.5070.objecter checking command 1 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6812/115952 conn(0x7f76e406e800 0x7f76e4070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:44.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter _get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 15 client.5070.objecter _session_command_op_remove -1 1 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter _get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 15 client.5070.objecter _session_command_op_assign 3 1 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 4 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 10 client.5070.objecter _send_command 1 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 --> v2:172.21.15.115:6812/115952 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76e4071280 con 0x7f76e406e800 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 20 client.5070.objecter dump_active .. 0 homeless 2023-09-30T01:08:44.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76f8088240 con 0x7f770013e040 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fffff700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6812/115952 conn(0x7f76e406e800 0x7f76e4070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fffff700 10 monclient: get_auth_request con 0x7f76e406e800 auth_method 0 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 10 client.5070.objecter ms_handle_connect 0x7f76e4068890 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f7704eb0700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6816/115635 conn(0x7f76e4068890 0x7f76e406ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76f40091e0 tx=0x7f76f4016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fcff9700 10 client.5070.objecter ms_handle_connect 0x7f76e406e800 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.844+0000 7f76fffff700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6812/115952 conn(0x7f76e406e800 0x7f76e4070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.845+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== osd.3 v2:172.21.15.115:6812/115952 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f76e4071280 con 0x7f76e406e800 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.845+0000 7f76fcff9700 10 client.5070.objecter ms_dispatch 0x7f7700001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.845+0000 7f76fcff9700 10 client.5070.objecter _finish_command 1 = system:0 2023-09-30T01:08:44.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.845+0000 7f76fcff9700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 4 2023-09-30T01:08:44.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.845+0000 7f76fcff9700 15 client.5070.objecter _session_command_op_remove 3 1 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 10 client.5070.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 15 client.5070.objecter _session_command_op_assign -1 2 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 20 client.5070.objecter _get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 20 client.5070.objecter _get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 15 client.5070.objecter _session_command_op_remove -1 2 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 20 client.5070.objecter get_session s=0x7f76e406e150 osd=3 3 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 15 client.5070.objecter _session_command_op_assign 3 2 2023-09-30T01:08:44.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 4 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 10 client.5070.objecter _send_command 2 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 --> v2:172.21.15.115:6812/115952 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7700136770 con 0x7f76e406e800 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f76fcff9700 1 -- 172.21.15.115:0/3343540665 <== osd.3 v2:172.21.15.115:6812/115952 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7700136770 con 0x7f76e406e800 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f76fcff9700 10 client.5070.objecter ms_dispatch 0x7f7700001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f76fcff9700 10 client.5070.objecter _finish_command 2 = system:0 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f76fcff9700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 4 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.858+0000 7f76fcff9700 15 client.5070.objecter _session_command_op_remove 3 2 2023-09-30T01:08:44.872 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:44.873 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:44.873 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:44.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 10 librados: watch_flush enter 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 10 librados: watch_flush exit 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 10 client.5070.objecter close_session for osd.3 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6812/115952 conn(0x7f76e406e800 msgr2=0x7f76e4070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6812/115952 conn(0x7f76e406e800 0x7f76e4070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 20 client.5070.objecter put_session s=0x7f76e406e150 osd=3 1 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 20 client.5070.objecter shutdown clearing up homeless session... 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 10 client.5070.objecter successfully canceled tick 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6816/115635 conn(0x7f76e4068890 msgr2=0x7f76e406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 --2- 172.21.15.115:0/3343540665 >> v2:172.21.15.115:6816/115635 conn(0x7f76e4068890 0x7f76e406ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76f40091e0 tx=0x7f76f4016000 comp rx=0 tx=0).stop 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 10 monclient: shutdown 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:44.875 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f770013e040 msgr2=0x7f77000a9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 --2- 172.21.15.115:0/3343540665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f770013e040 0x7f77000a9930 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f76f800c840 tx=0x7f76f8002620 comp rx=0 tx=0).stop 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f77056b1700 1 -- 172.21.15.115:0/3343540665 reap_dead start 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 shutdown_connections 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 >> 172.21.15.115:0/3343540665 conn(0x7f7700068e40 msgr2=0x7f7700137d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 shutdown_connections 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 -- 172.21.15.115:0/3343540665 wait complete. 2023-09-30T01:08:44.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.859+0000 7f7707114700 1 librados: shutdown 2023-09-30T01:08:44.876 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.993+0000 7eff1fdec700 10 monclient: get_monmap_and_config 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.993+0000 7eff1fdec700 10 monclient: build_initial_monmap 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.993+0000 7eff1fdec700 10 monclient: monmap: 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:44.994150+0000 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:44.994150+0000 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:44.993 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:44.994 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.993+0000 7eff1fdec700 1 Processor -- start 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.993+0000 7eff1fdec700 1 -- start start 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient: init 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff180ae8f0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 0x7eff180a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-b con 0x7eff180aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff181415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180a7040 con 0x7eff181415c0 2023-09-30T01:08:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180a57f0 con 0x7eff180ae8f0 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180a5480 con 0x7eff180aecc0 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1fdec700 10 monclient(hunting): authenticate will time out at 1386.444644s 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 0x7eff180a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1e389700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7eff181415c0 0x7eff180adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52818/0 (socket says 172.21.15.115:52818) 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1e389700 1 -- 172.21.15.115:0/91267234 learned_addr learned my addr 172.21.15.115:0/91267234 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): get_auth_request con 0x7eff180aecc0 auth_method 0 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient(hunting): handle_auth_done global_id 4784 payload 274 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 1 -- 172.21.15.115:0/91267234 >> v1:172.21.15.157:6789/0 conn(0x7eff180ae8f0 legacy=0x7eff180b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 1 -- 172.21.15.115:0/91267234 >> v1:172.21.15.115:6789/0 conn(0x7eff181415c0 legacy=0x7eff180adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1d387700 1 monclient: found mon.noname-b 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1d387700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.994+0000 7eff1cb86700 1 -- 172.21.15.115:0/91267234 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906695254 0 0) 0x7eff180a7040 con 0x7eff181415c0 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1d387700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1d387700 1 -- 172.21.15.115:0/91267234 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180a55c0 con 0x7eff180aecc0 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:45.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1d387700 1 --2- 172.21.15.115:0/91267234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 0x7eff180a5d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7eff14009910 tx=0x7eff140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63493a832c1ee94 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 1 -- 172.21.15.115:0/91267234 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1400b820 con 0x7eff180aecc0 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: dump: 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.006 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 1 -- 172.21.15.115:0/91267234 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1400b980 con 0x7eff180aecc0 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 5 monclient: authenticate success, global_id 4784 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 4 set_mon_vals no callback set 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 1 -- 172.21.15.115:0/91267234 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1400bd20 con 0x7eff180aecc0 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1cb86700 10 monclient: dump: 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 10 monclient: shutdown 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 -- 172.21.15.115:0/91267234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 msgr2=0x7eff180a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 --2- 172.21.15.115:0/91267234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 0x7eff180a5d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7eff14009910 tx=0x7eff140091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 -- 172.21.15.115:0/91267234 shutdown_connections 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 --2- 172.21.15.115:0/91267234 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff180aecc0 0x7eff180a5d10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 -- 172.21.15.115:0/91267234 >> 172.21.15.115:0/91267234 conn(0x7eff1806a460 msgr2=0x7eff180630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 -- 172.21.15.115:0/91267234 shutdown_connections 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.995+0000 7eff1fdec700 1 -- 172.21.15.115:0/91267234 wait complete. 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 10 monclient: monmap: 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.009 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 librados: starting msgr at 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 librados: starting objecter 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 Processor -- start 2023-09-30T01:08:45.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 -- start start 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 librados: setting wanted keys 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 1 librados: calling monclient init 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.996+0000 7eff1fdec700 10 monclient: init 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-a con 0x7eff180ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff180aecc0 0x7eff1813e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-b con 0x7eff180aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 0x7eff180a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): picked mon.noname-c con 0x7eff181415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180a9e70 con 0x7eff180ae8f0 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180a9b70 con 0x7eff180aecc0 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180ab6c0 con 0x7eff181415c0 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1fdec700 10 monclient(hunting): authenticate will time out at 1386.447459s 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 0x7eff180a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56918/0 (socket says 172.21.15.115:56918) 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 1 -- 172.21.15.115:0/1996194080 learned_addr learned my addr 172.21.15.115:0/1996194080 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): get_auth_request con 0x7eff180ae8f0 auth_method 0 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): get_auth_request con 0x7eff181415c0 auth_method 0 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1d387700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff180aecc0 0x7eff1813e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 -- 172.21.15.115:0/1996194080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff180aecc0 msgr2=0x7eff1813e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff180aecc0 0x7eff1813e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 -- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 msgr2=0x7eff180aa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 -- 172.21.15.115:0/1996194080 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14009420 con 0x7eff181415c0 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1db88700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff1e389700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 0x7eff180a7790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7eff1000cbc0 tx=0x7eff100018f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff09ffb700 10 client.?.objecter ms_handle_connect 0x7eff181415c0 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.997+0000 7eff09ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000a7f0 con 0x7eff181415c0 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: dump: 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.015 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 5 monclient: authenticate success, global_id 4787 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1000a970 con 0x7eff181415c0 2023-09-30T01:08:45.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 10 monclient: _renew_subs 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1813ede0 con 0x7eff181415c0 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000ad30 con 0x7eff181415c0 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff09ffb700 10 monclient: dump: 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.017 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff0bfff700 4 set_mon_vals no callback set 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 10 client.4787.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 10 monclient: _renew_subs 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff180a96e0 con 0x7eff181415c0 2023-09-30T01:08:45.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.998+0000 7eff1fdec700 1 librados: init done 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.999+0000 7eff1fdec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.999+0000 7eff1fdec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.999+0000 7eff1fdec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:44.999+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff180aa020 con 0x7eff181415c0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7eff10007300 con 0x7eff181415c0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 1 --2- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 0x7efefc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7eff100840b0 con 0x7eff181415c0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 10 client.4787.objecter ms_dispatch 0x7eff18001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 3 client.4787.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 3 client.4787.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff1db88700 1 --2- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 0x7efefc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff1db88700 10 monclient: get_auth_request con 0x7efefc068890 auth_method 0 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 20 client.4787.objecter dump_active .. 0 homeless 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff09ffb700 10 client.4787.objecter ms_handle_connect 0x7efefc068890 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.000+0000 7eff1db88700 1 --2- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 0x7efefc06ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff0c007fd0 tx=0x7eff0c007f60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.004+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff180aa020 con 0x7eff181415c0 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.004+0000 7eff09ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.004+0000 7eff09ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:45.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.186+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7eff180aa020 con 0x7efefc068890 2023-09-30T01:08:45.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.186+0000 7eff09ffb700 1 -- 172.21.15.115:0/1996194080 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7eff180aa020 con 0x7efefc068890 2023-09-30T01:08:45.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 10 librados: watch_flush enter 2023-09-30T01:08:45.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 10 librados: watch_flush exit 2023-09-30T01:08:45.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 20 client.4787.objecter shutdown clearing up homeless session... 2023-09-30T01:08:45.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 10 client.4787.objecter successfully canceled tick 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 msgr2=0x7efefc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 0x7efefc06ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff0c007fd0 tx=0x7eff0c007f60 comp rx=0 tx=0).stop 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 10 monclient: shutdown 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 msgr2=0x7eff180a7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 0x7eff180a7790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7eff1000cbc0 tx=0x7eff100018f0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 shutdown_connections 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> v2:172.21.15.115:6816/115635 conn(0x7efefc068890 0x7efefc06ad40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7eff181415c0 0x7eff180a7790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ae8f0 0x7eff180aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 --2- 172.21.15.115:0/1996194080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff180aecc0 0x7eff1813e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.188+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 >> 172.21.15.115:0/1996194080 conn(0x7eff1806a460 msgr2=0x7eff18144930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.189+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 shutdown_connections 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.189+0000 7eff1fdec700 1 -- 172.21.15.115:0/1996194080 wait complete. 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.189+0000 7eff1fdec700 1 librados: shutdown 2023-09-30T01:08:45.191 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T01:08:45.200 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T01:08:45.200 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:45.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.320+0000 7f8ee41ee700 10 monclient: get_monmap_and_config 2023-09-30T01:08:45.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.320+0000 7f8ee41ee700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.320+0000 7f8ee41ee700 10 monclient: monmap: 2023-09-30T01:08:45.320 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:45.321388+0000 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:45.321388+0000 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:45.321 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:45.322 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:45.322 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.320+0000 7f8ee41ee700 1 Processor -- start 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 -- start start 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient: init 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc0a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-b con 0x7f8edc0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8edc13e040 0x7f8edc13e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-a con 0x7f8edc13e040 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8edc141590 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc0a8220 con 0x7f8edc141590 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc0a6dc0 con 0x7f8edc13e040 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0a7d20 con 0x7f8edc0a5640 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee41ee700 10 monclient(hunting): authenticate will time out at 1386.771824s 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8edc13e040 0x7f8edc13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8edc13e040 0x7f8edc13e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56920/0 (socket says 172.21.15.115:56920) 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 1 -- 172.21.15.115:0/1957112343 learned_addr learned my addr 172.21.15.115:0/1957112343 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1f8a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc0a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 10 monclient(hunting): get_auth_request con 0x7f8edc13e040 auth_method 0 2023-09-30T01:08:45.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.321+0000 7f8ee1789700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5640 auth_method 0 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient(hunting): handle_auth_done global_id 5079 payload 274 2023-09-30T01:08:45.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 -- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 msgr2=0x7f8edc0a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc0a74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 -- 172.21.15.115:0/1957112343 >> v1:172.21.15.157:6789/0 conn(0x7f8edc141590 legacy=0x7f8edc1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 monclient: found mon.noname-a 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 -- 172.21.15.115:0/1957112343 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0a5a10 con 0x7f8edc13e040 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1f8a700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc0a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee1789700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc13e040 0x7f8edc13e490 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80099b0 tx=0x7f8ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65173f9c1e46e708 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 1 -- 172.21.15.115:0/1957112343 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8009ea0 con 0x7f8edc13e040 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: dump: 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.338 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.339 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.339 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee41ee700 5 monclient: authenticate success, global_id 5079 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee41ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 1 -- 172.21.15.115:0/1957112343 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed800b860 con 0x7f8edc13e040 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee41ee700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee41ee700 4 set_mon_vals no callback set 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 1 -- 172.21.15.115:0/1957112343 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800bc00 con 0x7f8edc13e040 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.322+0000 7f8ee0f88700 10 monclient: dump: 2023-09-30T01:08:45.341 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.342 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 10 monclient: shutdown 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 -- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc13e040 msgr2=0x7f8edc13e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc13e040 0x7f8edc13e490 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80099b0 tx=0x7f8ed8009250 comp rx=0 tx=0).stop 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 -- 172.21.15.115:0/1957112343 shutdown_connections 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc0a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/1957112343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc13e040 0x7f8edc13e490 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 -- 172.21.15.115:0/1957112343 >> 172.21.15.115:0/1957112343 conn(0x7f8edc068e40 msgr2=0x7f8edc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 -- 172.21.15.115:0/1957112343 shutdown_connections 2023-09-30T01:08:45.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 -- 172.21.15.115:0/1957112343 wait complete. 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 10 monclient: monmap: 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:45.344 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 librados: starting msgr at 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.323+0000 7f8ee41ee700 1 librados: starting objecter 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 Processor -- start 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 -- start start 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 librados: setting wanted keys 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 librados: calling monclient init 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient: init 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:45.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc13fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-c con 0x7f8edc0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8edc13e040 0x7f8edc1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-b con 0x7f8edc13e040 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc141590 0x7f8edc0a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): picked mon.noname-a con 0x7f8edc141590 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc13f770 con 0x7f8edc141590 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc140710 con 0x7f8edc13e040 2023-09-30T01:08:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc1408d0 con 0x7f8edc0a5640 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee41ee700 10 monclient(hunting): authenticate will time out at 1386.774699s 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc13fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc13fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36154/0 (socket says 172.21.15.115:36154) 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc141590 0x7f8edc0a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 1 -- 172.21.15.115:0/2561374275 learned_addr learned my addr 172.21.15.115:0/2561374275 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5640 auth_method 0 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): get_auth_request con 0x7f8edc141590 auth_method 0 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee278b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1789700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8edc13e040 0x7f8edc1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.324+0000 7f8ee1f8a700 10 monclient(hunting): handle_auth_done global_id 4793 payload 824 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 -- 172.21.15.115:0/2561374275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8edc13e040 msgr2=0x7f8edc1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8edc13e040 0x7f8edc1401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 -- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc141590 msgr2=0x7f8edc0a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc141590 0x7f8edc0a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 -- 172.21.15.115:0/2561374275 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc009420 con 0x7f8edc0a5640 2023-09-30T01:08:45.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1789700 10 monclient: get_auth_request con 0x7f8edc13e040 auth_method 0 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1789700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8edc13e040 0x7f8edc1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee278b700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8edc141590 0x7f8edc0a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee1f8a700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc13fc90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00ec80 tx=0x7f8ecc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 client.?.objecter ms_handle_connect 0x7f8edc0a5640 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00c8f0 con 0x7f8edc0a5640 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: dump: 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.350 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc00ca50 con 0x7f8edc0a5640 2023-09-30T01:08:45.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 5 monclient: authenticate success, global_id 4793 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 10 monclient: _renew_subs 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed80094f0 con 0x7f8edc0a5640 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00cdd0 con 0x7f8edc0a5640 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed1ffb700 10 monclient: dump: 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.352 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 10 client.4793.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 10 monclient: _renew_subs 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ed3fff700 4 set_mon_vals no callback set 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc0a96f0 con 0x7f8edc0a5640 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.325+0000 7f8ee41ee700 1 librados: init done 2023-09-30T01:08:45.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 10 client.4793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 15 client.4793.objecter _session_command_op_assign -1 1 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 20 client.4793.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 10 client.4793.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8edc0b04d0 con 0x7f8edc0a5640 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.326+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8ecc013680 con 0x7f8edc0a5640 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.115:6816/115635 conn(0x7f8ec0068890 0x7f8ec006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8ecc084560 con 0x7f8edc0a5640 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 10 client.4793.objecter ms_dispatch 0x7f8edc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:45.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 3 client.4793.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 3 client.4793.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ee1789700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.115:6816/115635 conn(0x7f8ec0068890 0x7f8ec006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ee1789700 10 monclient: get_auth_request con 0x7f8ec0068890 auth_method 0 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 10 client.4793.objecter checking command 1 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.157:6800/115846 conn(0x7f8ec006e800 0x7f8ec0070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter _get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 15 client.4793.objecter _session_command_op_remove -1 1 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter _get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 15 client.4793.objecter _session_command_op_assign 4 1 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 4 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 10 client.4793.objecter _send_command 1 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 --> v2:172.21.15.157:6800/115846 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ec0071280 con 0x7f8ec006e800 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 20 client.4793.objecter dump_active .. 0 homeless 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ecc084930 con 0x7f8edc0a5640 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ee1789700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.115:6816/115635 conn(0x7f8ec0068890 0x7f8ec006ad40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80099b0 tx=0x7f8ed8017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ed1ffb700 10 client.4793.objecter ms_handle_connect 0x7f8ec0068890 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ee278b700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.157:6800/115846 conn(0x7f8ec006e800 0x7f8ec0070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.327+0000 7f8ee278b700 10 monclient: get_auth_request con 0x7f8ec006e800 auth_method 0 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 10 client.4793.objecter ms_handle_connect 0x7f8ec006e800 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ee278b700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.157:6800/115846 conn(0x7f8ec006e800 0x7f8ec0070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== osd.4 v2:172.21.15.157:6800/115846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8ec0071280 con 0x7f8ec006e800 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 10 client.4793.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 10 client.4793.objecter _finish_command 1 = system:0 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 4 2023-09-30T01:08:45.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.328+0000 7f8ed1ffb700 15 client.4793.objecter _session_command_op_remove 4 1 2023-09-30T01:08:45.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 10 client.4793.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:45.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 15 client.4793.objecter _session_command_op_assign -1 2 2023-09-30T01:08:45.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 20 client.4793.objecter _get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 20 client.4793.objecter _get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 15 client.4793.objecter _session_command_op_remove -1 2 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 20 client.4793.objecter get_session s=0x7f8ec006e150 osd=4 3 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 15 client.4793.objecter _session_command_op_assign 4 2 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 4 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 10 client.4793.objecter _send_command 2 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 --> v2:172.21.15.157:6800/115846 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8edc072960 con 0x7f8ec006e800 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ed1ffb700 1 -- 172.21.15.115:0/2561374275 <== osd.4 v2:172.21.15.157:6800/115846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8edc072960 con 0x7f8ec006e800 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ed1ffb700 10 client.4793.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:45.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ed1ffb700 10 client.4793.objecter _finish_command 2 = system:0 2023-09-30T01:08:45.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ed1ffb700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 4 2023-09-30T01:08:45.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.342+0000 7f8ed1ffb700 15 client.4793.objecter _session_command_op_remove 4 2 2023-09-30T01:08:45.360 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:45.360 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:45.360 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 10 librados: watch_flush enter 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 10 librados: watch_flush exit 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 10 client.4793.objecter close_session for osd.4 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 >> v2:172.21.15.157:6800/115846 conn(0x7f8ec006e800 msgr2=0x7f8ec0070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.157:6800/115846 conn(0x7f8ec006e800 0x7f8ec0070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 20 client.4793.objecter put_session s=0x7f8ec006e150 osd=4 1 2023-09-30T01:08:45.362 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 20 client.4793.objecter shutdown clearing up homeless session... 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 10 client.4793.objecter successfully canceled tick 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 >> v2:172.21.15.115:6816/115635 conn(0x7f8ec0068890 msgr2=0x7f8ec006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/2561374275 >> v2:172.21.15.115:6816/115635 conn(0x7f8ec0068890 0x7f8ec006ad40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80099b0 tx=0x7f8ed8017000 comp rx=0 tx=0).stop 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 10 monclient: shutdown 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 msgr2=0x7f8edc13fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 --2- 172.21.15.115:0/2561374275 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f8edc0a5640 0x7f8edc13fc90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00ec80 tx=0x7f8ecc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee278b700 1 -- 172.21.15.115:0/2561374275 reap_dead start 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 shutdown_connections 2023-09-30T01:08:45.363 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 >> 172.21.15.115:0/2561374275 conn(0x7f8edc068e40 msgr2=0x7f8edc0afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 shutdown_connections 2023-09-30T01:08:45.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 -- 172.21.15.115:0/2561374275 wait complete. 2023-09-30T01:08:45.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.343+0000 7f8ee41ee700 1 librados: shutdown 2023-09-30T01:08:45.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 10 monclient: get_monmap_and_config 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 10 monclient: monmap: 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:45.483143+0000 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:45.483143+0000 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:45.483 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 1 Processor -- start 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 1 -- start start 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.482+0000 7f50ebd17700 10 monclient: init 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f50e40ae8f0 0x7f50e40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-c con 0x7f50e40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-b con 0x7f50e40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50e41415c0 0x7f50e40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-a con 0x7f50e41415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40a7040 con 0x7f50e41415c0 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40b19b0 con 0x7f50e40ae8f0 2023-09-30T01:08:45.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40a57f0 con 0x7f50e40aecc0 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ebd17700 10 monclient(hunting): authenticate will time out at 1386.933614s 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50e41415c0 0x7f50e40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f50e41415c0 0x7f50e40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56938/0 (socket says 172.21.15.115:56938) 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36168/0 (socket says 172.21.15.115:36168) 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 1 -- 172.21.15.115:0/3522882918 learned_addr learned my addr 172.21.15.115:0/3522882918 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): get_auth_request con 0x7f50e41415c0 auth_method 0 2023-09-30T01:08:45.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): get_auth_request con 0x7f50e40aecc0 auth_method 0 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 1 --2- 172.21.15.115:0/3522882918 >> v2:172.21.15.157:3300/0 conn(0x7f50e40ae8f0 0x7f50e40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): get_auth_request con 0x7f50e40ae8f0 auth_method 0 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.483+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_done global_id 5085 payload 274 2023-09-30T01:08:45.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 -- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 msgr2=0x7f50e40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 -- 172.21.15.115:0/3522882918 >> v2:172.21.15.157:3300/0 conn(0x7f50e40ae8f0 msgr2=0x7f50e40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 --2- 172.21.15.115:0/3522882918 >> v2:172.21.15.157:3300/0 conn(0x7f50e40ae8f0 0x7f50e40a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 monclient: found mon.noname-a 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 -- 172.21.15.115:0/3522882918 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40a5480 con 0x7f50e41415c0 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e92b2700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ea2b4700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40adb60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f50e40b59c0 tx=0x7f50dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a153ca7682fa883 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 1 -- 172.21.15.115:0/3522882918 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00e070 con 0x7f50e41415c0 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: dump: 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.500 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 5 monclient: authenticate success, global_id 5085 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 1 -- 172.21.15.115:0/3522882918 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc00b970 con 0x7f50e41415c0 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 4 set_mon_vals no callback set 2023-09-30T01:08:45.501 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 1 -- 172.21.15.115:0/3522882918 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0092e0 con 0x7f50e41415c0 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50e8ab1700 10 monclient: dump: 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.502 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 10 monclient: shutdown 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 -- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 msgr2=0x7f50e40adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40adb60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f50e40b59c0 tx=0x7f50dc009840 comp rx=0 tx=0).stop 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 -- 172.21.15.115:0/3522882918 shutdown_connections 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40adb60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 --2- 172.21.15.115:0/3522882918 >> v2:172.21.15.157:3300/0 conn(0x7f50e40ae8f0 0x7f50e40a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.503 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 --2- 172.21.15.115:0/3522882918 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.484+0000 7f50ebd17700 1 -- 172.21.15.115:0/3522882918 >> 172.21.15.115:0/3522882918 conn(0x7f50e406a460 msgr2=0x7f50e40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 -- 172.21.15.115:0/3522882918 shutdown_connections 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 -- 172.21.15.115:0/3522882918 wait complete. 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 10 monclient: monmap: 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.504 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 librados: starting msgr at 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 librados: starting objecter 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 Processor -- start 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 -- start start 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 librados: setting wanted keys 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 1 librados: calling monclient init 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.485+0000 7f50ebd17700 10 monclient: init 2023-09-30T01:08:45.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e40ae8f0 0x7f50e413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-b con 0x7f50e40ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-c con 0x7f50e40aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): picked mon.noname-a con 0x7f50e41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40ab490 con 0x7f50e41415c0 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40a9220 con 0x7f50e40ae8f0 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e413ecf0 con 0x7f50e40aecc0 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ebd17700 10 monclient(hunting): authenticate will time out at 1386.936479s 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56954/0 (socket says 172.21.15.115:56954) 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36178/0 (socket says 172.21.15.115:36178) 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 1 -- 172.21.15.115:0/2241871685 learned_addr learned my addr 172.21.15.115:0/2241871685 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): get_auth_request con 0x7f50e41415c0 auth_method 0 2023-09-30T01:08:45.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): get_auth_request con 0x7f50e40aecc0 auth_method 0 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e40ae8f0 0x7f50e413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): get_auth_request con 0x7f50e40ae8f0 auth_method 0 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e9ab3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient(hunting): handle_auth_done global_id 4799 payload 824 2023-09-30T01:08:45.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 -- 172.21.15.115:0/2241871685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e40ae8f0 msgr2=0x7f50e413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e40ae8f0 0x7f50e413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 -- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 msgr2=0x7f50e40a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50e92b2700 1 -- 172.21.15.115:0/2241871685 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e0009420 con 0x7f50e40aecc0 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.486+0000 7f50ea2b4700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50e92b2700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50e000ecb0 tx=0x7f50e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 client.?.objecter ms_handle_connect 0x7f50e40aecc0 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e000c850 con 0x7f50e40aecc0 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: dump: 2023-09-30T01:08:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 5 monclient: authenticate success, global_id 4799 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e000c9b0 con 0x7f50e40aecc0 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 10 monclient: _renew_subs 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc009ae0 con 0x7f50e40aecc0 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e000cd50 con 0x7f50e40aecc0 2023-09-30T01:08:45.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50d9ffb700 10 monclient: dump: 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50dbfff700 4 set_mon_vals no callback set 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 10 client.4799.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 10 monclient: _renew_subs 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e40a7730 con 0x7f50e40aecc0 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.487+0000 7f50ebd17700 1 librados: init done 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.488+0000 7f50ebd17700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.488+0000 7f50ebd17700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.488+0000 7f50ebd17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.488+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40a9720 con 0x7f50e40aecc0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.488+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f50e0013680 con 0x7f50e40aecc0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 1 --2- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 0x7f50c806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f50e0084150 con 0x7f50e40aecc0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 10 client.4799.objecter ms_dispatch 0x7f50e4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 3 client.4799.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 3 client.4799.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50e9ab3700 1 --2- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 0x7f50c806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50e9ab3700 10 monclient: get_auth_request con 0x7f50c8068890 auth_method 0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 20 client.4799.objecter dump_active .. 0 homeless 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50d9ffb700 10 client.4799.objecter ms_handle_connect 0x7f50c8068890 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.489+0000 7f50e9ab3700 1 --2- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 0x7f50c806ad40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009db0 tx=0x7f50d40091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.493+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50e0051970 con 0x7f50e40aecc0 2023-09-30T01:08:45.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.493+0000 7f50d9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.493+0000 7f50d9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:45.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.674+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f50e4072960 con 0x7f50c8068890 2023-09-30T01:08:45.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.674+0000 7f50d9ffb700 1 -- 172.21.15.115:0/2241871685 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f50e4072960 con 0x7f50c8068890 2023-09-30T01:08:45.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 10 librados: watch_flush enter 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 10 librados: watch_flush exit 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 20 client.4799.objecter shutdown clearing up homeless session... 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 10 client.4799.objecter successfully canceled tick 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 msgr2=0x7f50c806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 0x7f50c806ad40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009db0 tx=0x7f50d40091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 10 monclient: shutdown 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 msgr2=0x7f50e40a8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50e000ecb0 tx=0x7f50e00091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 shutdown_connections 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> v2:172.21.15.115:6816/115635 conn(0x7f50c8068890 0x7f50c806ad40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e41415c0 0x7f50e40a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e40ae8f0 0x7f50e413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 --2- 172.21.15.115:0/2241871685 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f50e40aecc0 0x7f50e40a8ce0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 >> 172.21.15.115:0/2241871685 conn(0x7f50e406a460 msgr2=0x7f50e4144710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 shutdown_connections 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 -- 172.21.15.115:0/2241871685 wait complete. 2023-09-30T01:08:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.676+0000 7f50ebd17700 1 librados: shutdown 2023-09-30T01:08:45.679 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T01:08:45.687 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T01:08:45.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:45.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.808+0000 7f2d83b71700 10 monclient: get_monmap_and_config 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.808+0000 7f2d83b71700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.808+0000 7f2d83b71700 10 monclient: monmap: 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:45.809489+0000 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:45.809489+0000 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:45.809 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:45.810 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:45.810 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:45.810 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 Processor -- start 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 -- start start 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient: init 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d7c0a5640 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d7c13e040 0x7f2d7c13e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-c con 0x7f2d7c13e040 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:45.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2d7c13acd0 0x7f2d7c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-a con 0x7f2d7c13acd0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c0a8220 con 0x7f2d7c0a5640 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0a6dc0 con 0x7f2d7c13acd0 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0a7d20 con 0x7f2d7c13e040 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.809+0000 7f2d83b71700 10 monclient(hunting): authenticate will time out at 1387.259906s 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2d7c13acd0 0x7f2d7c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f2d7c13acd0 0x7f2d7c13b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56970/0 (socket says 172.21.15.115:56970) 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 -- 172.21.15.115:0/584161185 learned_addr learned my addr 172.21.15.115:0/584161185 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): get_auth_request con 0x7f2d7c13acd0 auth_method 0 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8110c700 1 --2- 172.21.15.115:0/584161185 >> v2:172.21.15.157:3300/0 conn(0x7f2d7c13e040 0x7f2d7c13e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8110c700 10 monclient(hunting): get_auth_request con 0x7f2d7c13e040 auth_method 0 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8110c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8110c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8110c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient(hunting): handle_auth_done global_id 5091 payload 274 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 -- 172.21.15.115:0/584161185 >> v2:172.21.15.157:3300/0 conn(0x7f2d7c13e040 msgr2=0x7f2d7c13e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 --2- 172.21.15.115:0/584161185 >> v2:172.21.15.157:3300/0 conn(0x7f2d7c13e040 0x7f2d7c13e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 -- 172.21.15.115:0/584161185 >> v1:172.21.15.157:6789/0 conn(0x7f2d7c0a5640 legacy=0x7f2d7c0a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 monclient: found mon.noname-a 2023-09-30T01:08:45.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 -- 172.21.15.115:0/584161185 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c0a5a10 con 0x7f2d7c13acd0 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8210e700 1 --2- 172.21.15.115:0/584161185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c13b150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0b1ba0 tx=0x7f2d74009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f723bc26bc3ebf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 1 -- 172.21.15.115:0/584161185 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7400e070 con 0x7f2d7c13acd0 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: dump: 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.824 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 1 -- 172.21.15.115:0/584161185 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7400ba50 con 0x7f2d7c13acd0 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d83b71700 5 monclient: authenticate success, global_id 5091 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d83b71700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 1 -- 172.21.15.115:0/584161185 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740092e0 con 0x7f2d7c13acd0 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d83b71700 4 set_mon_vals no callback set 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.810+0000 7f2d8090b700 10 monclient: dump: 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 10 monclient: shutdown 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 -- 172.21.15.115:0/584161185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13acd0 msgr2=0x7f2d7c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 --2- 172.21.15.115:0/584161185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c13b150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0b1ba0 tx=0x7f2d74009840 comp rx=0 tx=0).stop 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 -- 172.21.15.115:0/584161185 shutdown_connections 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 --2- 172.21.15.115:0/584161185 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c13b150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 --2- 172.21.15.115:0/584161185 >> v2:172.21.15.157:3300/0 conn(0x7f2d7c13e040 0x7f2d7c13e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 -- 172.21.15.115:0/584161185 >> 172.21.15.115:0/584161185 conn(0x7f2d7c068e40 msgr2=0x7f2d7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 -- 172.21.15.115:0/584161185 shutdown_connections 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 -- 172.21.15.115:0/584161185 wait complete. 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 10 monclient: monmap: 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.811+0000 7f2d83b71700 1 librados: starting msgr at 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 librados: starting objecter 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 Processor -- start 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 -- start start 2023-09-30T01:08:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 librados: setting wanted keys 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 librados: calling monclient init 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient: init 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2d7c0a5640 0x7f2d7c0aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-c con 0x7f2d7c0a5640 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-b con 0x7f2d7c13acd0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 0x7f2d7c0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): picked mon.noname-a con 0x7f2d7c13e040 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b4910 con 0x7f2d7c13e040 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0ac8e0 con 0x7f2d7c13acd0 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0acbc0 con 0x7f2d7c0a5640 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d83b71700 10 monclient(hunting): authenticate will time out at 1387.262682s 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 0x7f2d7c0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2d7c0a5640 0x7f2d7c0aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 0x7f2d7c0b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56974/0 (socket says 172.21.15.115:56974) 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 1 -- 172.21.15.115:0/1598502099 learned_addr learned my addr 172.21.15.115:0/1598502099 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): get_auth_request con 0x7f2d7c13e040 auth_method 0 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): get_auth_request con 0x7f2d7c0a5640 auth_method 0 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8210e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8190d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): get_auth_request con 0x7f2d7c13acd0 auth_method 0 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.812+0000 7f2d8110c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 10 monclient(hunting): handle_auth_done global_id 4802 payload 824 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 -- 172.21.15.115:0/1598502099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d7c13acd0 msgr2=0x7f2d7c0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d7c13acd0 0x7f2d7c0b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 -- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 msgr2=0x7f2d7c0b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 0x7f2d7c0b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 -- 172.21.15.115:0/1598502099 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74009ae0 con 0x7f2d7c0a5640 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8210e700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d7c13e040 0x7f2d7c0b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d8190d700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2d7c0a5640 0x7f2d7c0aa7f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c590 tx=0x7f2d78007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 client.?.objecter ms_handle_connect 0x7f2d7c0a5640 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78011590 con 0x7f2d7c0a5640 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: dump: 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d78011710 con 0x7f2d7c0a5640 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 5 monclient: authenticate success, global_id 4802 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 10 monclient: _renew_subs 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c0b3820 con 0x7f2d7c0a5640 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78011ad0 con 0x7f2d7c0a5640 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d71ffb700 10 monclient: dump: 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d73fff700 4 set_mon_vals no callback set 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 10 client.4802.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 10 monclient: _renew_subs 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c0b4630 con 0x7f2d7c0a5640 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.813+0000 7f2d83b71700 1 librados: init done 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 10 client.4802.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 15 client.4802.objecter _session_command_op_assign -1 1 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 20 client.4802.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 10 client.4802.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d7c072960 con 0x7f2d7c0a5640 2023-09-30T01:08:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.814+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2d7801b3b0 con 0x7f2d7c0a5640 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.115:6816/115635 conn(0x7f2d60068890 0x7f2d6006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8110c700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.115:6816/115635 conn(0x7f2d60068890 0x7f2d6006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2d78083250 con 0x7f2d7c0a5640 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 client.4802.objecter ms_dispatch 0x7f2d7c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 3 client.4802.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 3 client.4802.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8110c700 10 monclient: get_auth_request con 0x7f2d60068890 auth_method 0 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 client.4802.objecter checking command 1 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.157:6804/115847 conn(0x7f2d6006e800 0x7f2d60070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter _get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 15 client.4802.objecter _session_command_op_remove -1 1 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter _get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 15 client.4802.objecter _session_command_op_assign 5 1 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 4 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 client.4802.objecter _send_command 1 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 --> v2:172.21.15.157:6804/115847 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d60071280 con 0x7f2d6006e800 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 20 client.4802.objecter dump_active .. 0 homeless 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d78085890 con 0x7f2d7c0a5640 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 client.4802.objecter ms_handle_connect 0x7f2d60068890 2023-09-30T01:08:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8110c700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.115:6816/115635 conn(0x7f2d60068890 0x7f2d6006ad40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c13bf10 tx=0x7f2d6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8210e700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.157:6804/115847 conn(0x7f2d6006e800 0x7f2d60070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8210e700 10 monclient: get_auth_request con 0x7f2d6006e800 auth_method 0 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d71ffb700 10 client.4802.objecter ms_handle_connect 0x7f2d6006e800 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.815+0000 7f2d8210e700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.157:6804/115847 conn(0x7f2d6006e800 0x7f2d60070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.816+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== osd.5 v2:172.21.15.157:6804/115847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2d60071280 con 0x7f2d6006e800 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.816+0000 7f2d71ffb700 10 client.4802.objecter ms_dispatch 0x7f2d7c001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.816+0000 7f2d71ffb700 10 client.4802.objecter _finish_command 1 = system:0 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.816+0000 7f2d71ffb700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 4 2023-09-30T01:08:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.816+0000 7f2d71ffb700 15 client.4802.objecter _session_command_op_remove 5 1 2023-09-30T01:08:45.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 client.4802.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:45.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 15 client.4802.objecter _session_command_op_assign -1 2 2023-09-30T01:08:45.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter _get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter _get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 15 client.4802.objecter _session_command_op_remove -1 2 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter get_session s=0x7f2d6006e150 osd=5 3 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 15 client.4802.objecter _session_command_op_assign 5 2 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 4 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 client.4802.objecter _send_command 2 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 --> v2:172.21.15.157:6804/115847 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2d7c0aec70 con 0x7f2d6006e800 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d71ffb700 1 -- 172.21.15.115:0/1598502099 <== osd.5 v2:172.21.15.157:6804/115847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2d7c0aec70 con 0x7f2d6006e800 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d71ffb700 10 client.4802.objecter ms_dispatch 0x7f2d7c001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d71ffb700 10 client.4802.objecter _finish_command 2 = system:0 2023-09-30T01:08:45.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d71ffb700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 4 2023-09-30T01:08:45.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d71ffb700 15 client.4802.objecter _session_command_op_remove 5 2 2023-09-30T01:08:45.845 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:45.845 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:45.845 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:45.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 librados: watch_flush enter 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 librados: watch_flush exit 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 client.4802.objecter close_session for osd.5 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 >> v2:172.21.15.157:6804/115847 conn(0x7f2d6006e800 msgr2=0x7f2d60070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.157:6804/115847 conn(0x7f2d6006e800 0x7f2d60070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter put_session s=0x7f2d6006e150 osd=5 1 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 client.4802.objecter shutdown clearing up homeless session... 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 client.4802.objecter successfully canceled tick 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 >> v2:172.21.15.115:6816/115635 conn(0x7f2d60068890 msgr2=0x7f2d6006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 --2- 172.21.15.115:0/1598502099 >> v2:172.21.15.115:6816/115635 conn(0x7f2d60068890 0x7f2d6006ad40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c13bf10 tx=0x7f2d6c008040 comp rx=0 tx=0).stop 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 10 monclient: shutdown 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2d7c0a5640 msgr2=0x7f2d7c0aa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d83b71700 1 --2- 172.21.15.115:0/1598502099 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f2d7c0a5640 0x7f2d7c0aa7f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800c590 tx=0x7f2d78007ec0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.830+0000 7f2d8210e700 1 -- 172.21.15.115:0/1598502099 reap_dead start 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.831+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 shutdown_connections 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.831+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 >> 172.21.15.115:0/1598502099 conn(0x7f2d7c068e40 msgr2=0x7f2d7c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.831+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 shutdown_connections 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.831+0000 7f2d83b71700 1 -- 172.21.15.115:0/1598502099 wait complete. 2023-09-30T01:08:45.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.831+0000 7f2d83b71700 1 librados: shutdown 2023-09-30T01:08:45.851 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.968+0000 7f35c13ec700 10 monclient: get_monmap_and_config 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.968+0000 7f35c13ec700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.968+0000 7f35c13ec700 10 monclient: monmap: 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:45.969747+0000 2023-09-30T01:08:45.969 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:45.969747+0000 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 1 Processor -- start 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 1 -- start start 2023-09-30T01:08:45.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 10 monclient: init 2023-09-30T01:08:45.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-09-30T01:08:45.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f35bc0ae8f0 0x7f35bc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-c con 0x7f35bc0ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.969+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35bc0aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35bc1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0a7040 con 0x7f35bc1415c0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0a57f0 con 0x7f35bc0aecc0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc0a5480 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 10 monclient(hunting): authenticate will time out at 1387.420054s 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35bb7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f35bc1415c0 0x7f35bc0adb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52824/0 (socket says 172.21.15.115:52824) 2023-09-30T01:08:45.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35bb7fe700 1 -- 172.21.15.115:0/2074368599 learned_addr learned my addr 172.21.15.115:0/2074368599 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 --2- 172.21.15.115:0/2074368599 >> v2:172.21.15.157:3300/0 conn(0x7f35bc0ae8f0 0x7f35bc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): get_auth_request con 0x7f35bc0ae8f0 auth_method 0 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 1 -- 172.21.15.115:0/2074368599 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667017770 0 0) 0x7f35bc0a7040 con 0x7f35bc1415c0 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient(hunting): my global_id is 5097 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 1 -- 172.21.15.115:0/2074368599 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4003570 con 0x7f35bc1415c0 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient(hunting): handle_auth_done global_id 4846 payload 274 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 -- 172.21.15.115:0/2074368599 >> v1:172.21.15.157:6789/0 conn(0x7f35bc0aecc0 legacy=0x7f35bc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 -- 172.21.15.115:0/2074368599 >> v1:172.21.15.115:6789/0 conn(0x7f35bc1415c0 legacy=0x7f35bc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 -- 172.21.15.115:0/2074368599 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0a55c0 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35baffd700 1 --2- 172.21.15.115:0/2074368599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a5d10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f35b00098e0 tx=0x7f35b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e2d4a65662ad90a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 1 -- 172.21.15.115:0/2074368599 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000b640 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: dump: 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.983 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 5 monclient: authenticate success, global_id 4846 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:45.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 1 -- 172.21.15.115:0/2074368599 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000b7a0 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35c13ec700 4 set_mon_vals no callback set 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 1 -- 172.21.15.115:0/2074368599 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000bb40 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.970+0000 7f35b9ffb700 10 monclient: dump: 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.985 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 10 monclient: shutdown 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 -- 172.21.15.115:0/2074368599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0ae8f0 msgr2=0x7f35bc0a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 --2- 172.21.15.115:0/2074368599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a5d10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f35b00098e0 tx=0x7f35b00091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 -- 172.21.15.115:0/2074368599 shutdown_connections 2023-09-30T01:08:45.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 --2- 172.21.15.115:0/2074368599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a5d10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 -- 172.21.15.115:0/2074368599 >> 172.21.15.115:0/2074368599 conn(0x7f35bc06a460 msgr2=0x7f35bc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 -- 172.21.15.115:0/2074368599 shutdown_connections 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.971+0000 7f35c13ec700 1 -- 172.21.15.115:0/2074368599 wait complete. 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.972+0000 7f35c13ec700 10 monclient: build_initial_monmap 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.972+0000 7f35c13ec700 10 monclient: monmap: 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:45.987 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.972+0000 7f35c13ec700 1 librados: starting msgr at 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.972+0000 7f35c13ec700 1 librados: starting objecter 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.972+0000 7f35c13ec700 1 Processor -- start 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 -- start start 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 librados: setting wanted keys 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 librados: calling monclient init 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient: init 2023-09-30T01:08:45.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-a con 0x7f35bc0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0aecc0 0x7f35bc14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-b con 0x7f35bc0aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 0x7f35bc14e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): picked mon.noname-c con 0x7f35bc1415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc13d300 con 0x7f35bc0ae8f0 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0a98d0 con 0x7f35bc0aecc0 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0aa6f0 con 0x7f35bc1415c0 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35c13ec700 10 monclient(hunting): authenticate will time out at 1387.423785s 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56976/0 (socket says 172.21.15.115:56976) 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 0x7f35bc14e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 1 -- 172.21.15.115:0/1665986670 learned_addr learned my addr 172.21.15.115:0/1665986670 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 10 monclient(hunting): get_auth_request con 0x7f35bc0ae8f0 auth_method 0 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): get_auth_request con 0x7f35bc1415c0 auth_method 0 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35ba7fc700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0aecc0 0x7f35bc14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35ba7fc700 10 monclient(hunting): get_auth_request con 0x7f35bc0aecc0 auth_method 0 2023-09-30T01:08:45.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35ba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.973+0000 7f35ba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 -- 172.21.15.115:0/1665986670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0aecc0 msgr2=0x7f35bc14dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0aecc0 0x7f35bc14dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 -- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 msgr2=0x7f35bc0a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 monclient: found mon.noname-c 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 -- 172.21.15.115:0/1665986670 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0009420 con 0x7f35bc1415c0 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35baffd700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35bb7fe700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 0x7f35bc14e2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35b400cbf0 tx=0x7f35b4001a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 client.?.objecter ms_handle_connect 0x7f35bc1415c0 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:45.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400a870 con 0x7f35bc1415c0 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: dump: 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.994 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b400a9d0 con 0x7f35bc1415c0 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 5 monclient: authenticate success, global_id 4805 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 10 monclient: _renew_subs 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc0aa9d0 con 0x7f35bc1415c0 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400ad50 con 0x7f35bc1415c0 2023-09-30T01:08:45.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35a2ffd700 10 monclient: dump: 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:45.996 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 10 client.4805.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 10 monclient: _renew_subs 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc0a9380 con 0x7f35bc1415c0 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35b97fa700 4 set_mon_vals no callback set 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.974+0000 7f35c13ec700 1 librados: init done 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.975+0000 7f35c13ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.975+0000 7f35c13ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.975+0000 7f35c13ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:45.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.975+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc14dac0 con 0x7f35bc1415c0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.975+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f35b4007300 con 0x7f35bc1415c0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 1 --2- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 0x7f35a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35baffd700 1 --2- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 0x7f35a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f35b4084110 con 0x7f35bc1415c0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 10 client.4805.objecter ms_dispatch 0x7f35bc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35baffd700 10 monclient: get_auth_request con 0x7f35a8068890 auth_method 0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 3 client.4805.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 3 client.4805.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 20 client.4805.objecter dump_active .. 0 homeless 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35a2ffd700 10 client.4805.objecter ms_handle_connect 0x7f35a8068890 2023-09-30T01:08:45.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.976+0000 7f35baffd700 1 --2- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 0x7f35a806ad40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35b00093a0 tx=0x7f35b0017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:45.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.980+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35bc14dac0 con 0x7f35bc1415c0 2023-09-30T01:08:45.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.980+0000 7f35a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:45.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:45.980+0000 7f35a2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:46.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.161+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f35bc14dac0 con 0x7f35a8068890 2023-09-30T01:08:46.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.161+0000 7f35a2ffd700 1 -- 172.21.15.115:0/1665986670 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f35bc14dac0 con 0x7f35a8068890 2023-09-30T01:08:46.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 10 librados: watch_flush enter 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 10 librados: watch_flush exit 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 20 client.4805.objecter shutdown clearing up homeless session... 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 10 client.4805.objecter successfully canceled tick 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 msgr2=0x7f35a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 0x7f35a806ad40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35b00093a0 tx=0x7f35b0017000 comp rx=0 tx=0).stop 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 10 monclient: shutdown 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 msgr2=0x7f35bc14e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 0x7f35bc14e2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35b400cbf0 tx=0x7f35b4001a20 comp rx=0 tx=0).stop 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 shutdown_connections 2023-09-30T01:08:46.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> v2:172.21.15.115:6816/115635 conn(0x7f35a8068890 0x7f35a806ad40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f35bc1415c0 0x7f35bc14e2e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35bc0ae8f0 0x7f35bc0a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 --2- 172.21.15.115:0/1665986670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35bc0aecc0 0x7f35bc14dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 >> 172.21.15.115:0/1665986670 conn(0x7f35bc06a460 msgr2=0x7f35bc13ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 shutdown_connections 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 -- 172.21.15.115:0/1665986670 wait complete. 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.163+0000 7f35c13ec700 1 librados: shutdown 2023-09-30T01:08:46.165 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T01:08:46.175 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T01:08:46.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 10 monclient: get_monmap_and_config 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 10 monclient: monmap: 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:46.296025+0000 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:46.296025+0000 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.295 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:46.296 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:46.296 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:46.296 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:46.296 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:46.296 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 1 Processor -- start 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 1 -- start start 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.295+0000 7f22ceb57700 10 monclient: init 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22c80a5640 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f22c813e040 0x7f22c813e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-c con 0x7f22c813e040 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22c813acd0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c80a8220 con 0x7f22c813acd0 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c80a6dc0 con 0x7f22c80a5640 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80a7d20 con 0x7f22c813e040 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22ceb57700 10 monclient(hunting): authenticate will time out at 1387.746498s 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22cd0f4700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f22c813acd0 0x7f22c813b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52832/0 (socket says 172.21.15.115:52832) 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22cd0f4700 1 -- 172.21.15.115:0/4163713803 learned_addr learned my addr 172.21.15.115:0/4163713803 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 1 --2- 172.21.15.115:0/4163713803 >> v2:172.21.15.157:3300/0 conn(0x7f22c813e040 0x7f22c813e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): get_auth_request con 0x7f22c813e040 auth_method 0 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175097257 0 0) 0x7f22c80a8220 con 0x7f22c813acd0 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): my global_id is 5103 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b8003570 con 0x7f22c813acd0 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988813021 0 0) 0x7f22c80a6dc0 con 0x7f22c80a5640 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 10 monclient(hunting): my global_id is 4858 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c80a8220 con 0x7f22c80a5640 2023-09-30T01:08:46.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient(hunting): handle_auth_done global_id 4855 payload 274 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 1 -- 172.21.15.115:0/4163713803 >> v1:172.21.15.157:6789/0 conn(0x7f22c80a5640 legacy=0x7f22c80a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 1 -- 172.21.15.115:0/4163713803 >> v1:172.21.15.115:6789/0 conn(0x7f22c813acd0 legacy=0x7f22c813b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 1 monclient: found mon.noname-c 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.157:3300/0 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.296+0000 7f22c7fff700 1 -- 172.21.15.115:0/4163713803 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80a5a10 con 0x7f22c813e040 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c7fff700 1 --2- 172.21.15.115:0/4163713803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813e040 0x7f22c813e4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f22b40098e0 tx=0x7f22b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d51d97f9bf727cb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4011630 con 0x7f22c813e040 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: dump: 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b4011790 con 0x7f22c813e040 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 5 monclient: authenticate success, global_id 4855 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 4 set_mon_vals no callback set 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 1 -- 172.21.15.115:0/4163713803 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4011b30 con 0x7f22c813e040 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22c77fe700 10 monclient: dump: 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 10 monclient: shutdown 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 -- 172.21.15.115:0/4163713803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813e040 msgr2=0x7f22c813e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 --2- 172.21.15.115:0/4163713803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813e040 0x7f22c813e4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f22b40098e0 tx=0x7f22b40091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 -- 172.21.15.115:0/4163713803 shutdown_connections 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 --2- 172.21.15.115:0/4163713803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813e040 0x7f22c813e4c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 -- 172.21.15.115:0/4163713803 >> 172.21.15.115:0/4163713803 conn(0x7f22c8068e40 msgr2=0x7f22c80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 -- 172.21.15.115:0/4163713803 shutdown_connections 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.297+0000 7f22ceb57700 1 -- 172.21.15.115:0/4163713803 wait complete. 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.298+0000 7f22ceb57700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.298+0000 7f22ceb57700 10 monclient: monmap: 2023-09-30T01:08:46.314 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.298+0000 7f22ceb57700 1 librados: starting msgr at 2023-09-30T01:08:46.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.298+0000 7f22ceb57700 1 librados: starting objecter 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.298+0000 7f22ceb57700 1 Processor -- start 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 -- start start 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 librados: setting wanted keys 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 librados: calling monclient init 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient: init 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 0x7f22c80aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-a con 0x7f22c80a5640 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813acd0 0x7f22c8137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-b con 0x7f22c813acd0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:46.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f22c813e040 0x7f22c8138090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): picked mon.noname-c con 0x7f22c813e040 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80addc0 con 0x7f22c80a5640 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c81385d0 con 0x7f22c813acd0 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80ada90 con 0x7f22c813e040 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22ceb57700 10 monclient(hunting): authenticate will time out at 1387.749697s 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 0x7f22c80aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 0x7f22c80aacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:56992/0 (socket says 172.21.15.115:56992) 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 1 -- 172.21.15.115:0/2956883424 learned_addr learned my addr 172.21.15.115:0/2956883424 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): get_auth_request con 0x7f22c80a5640 auth_method 0 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813acd0 0x7f22c8137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f22c813e040 0x7f22c8138090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): get_auth_request con 0x7f22c813acd0 auth_method 0 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): get_auth_request con 0x7f22c813e040 auth_method 0 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cc8f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.299+0000 7f22cd0f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 10 monclient(hunting): handle_auth_done global_id 5106 payload 824 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 -- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f22c813e040 msgr2=0x7f22c8138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f22c813e040 0x7f22c8138090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 -- 172.21.15.115:0/2956883424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813acd0 msgr2=0x7f22c8137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22c813acd0 0x7f22c8137b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 monclient: found mon.noname-a 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 -- 172.21.15.115:0/2956883424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b4009420 con 0x7f22c80a5640 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22cc8f3700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 0x7f22c80aacb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00c790 tx=0x7f22bc007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 client.?.objecter ms_handle_connect 0x7f22c80a5640 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:46.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc011590 con 0x7f22c80a5640 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: dump: 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.322 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 5 monclient: authenticate success, global_id 5106 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc0116f0 con 0x7f22c80a5640 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 10 monclient: _renew_subs 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c80b3b00 con 0x7f22c80a5640 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc011a90 con 0x7f22c80a5640 2023-09-30T01:08:46.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c4ff9700 10 monclient: dump: 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.324 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22c6ffd700 4 set_mon_vals no callback set 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 10 client.5106.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 10 monclient: _renew_subs 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c80ae6e0 con 0x7f22c80a5640 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.300+0000 7f22ceb57700 1 librados: init done 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 10 client.5106.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 15 client.5106.objecter _session_command_op_assign -1 1 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 20 client.5106.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 10 client.5106.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.325 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22c80ae990 con 0x7f22c80a5640 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.301+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f22bc01b3b0 con 0x7f22c80a5640 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.115:6816/115635 conn(0x7f22ac068890 0x7f22ac06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f22bc0833b0 con 0x7f22c80a5640 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 client.5106.objecter ms_dispatch 0x7f22c8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 3 client.5106.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 3 client.5106.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c7fff700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.115:6816/115635 conn(0x7f22ac068890 0x7f22ac06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c7fff700 10 monclient: get_auth_request con 0x7f22ac068890 auth_method 0 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 client.5106.objecter checking command 1 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.157:6808/115848 conn(0x7f22ac06e800 0x7f22ac070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter _get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 15 client.5106.objecter _session_command_op_remove -1 1 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter _get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 15 client.5106.objecter _session_command_op_assign 6 1 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 4 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 client.5106.objecter _send_command 1 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 --> v2:172.21.15.157:6808/115848 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22ac071280 con 0x7f22ac06e800 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 20 client.5106.objecter dump_active .. 0 homeless 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22bc085990 con 0x7f22c80a5640 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:46.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22cd0f4700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.157:6808/115848 conn(0x7f22ac06e800 0x7f22ac070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22cd0f4700 10 monclient: get_auth_request con 0x7f22ac06e800 auth_method 0 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 client.5106.objecter ms_handle_connect 0x7f22ac068890 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c7fff700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.115:6816/115635 conn(0x7f22ac068890 0x7f22ac06ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22b40093f0 tx=0x7f22b4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22c4ff9700 10 client.5106.objecter ms_handle_connect 0x7f22ac06e800 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.302+0000 7f22cd0f4700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.157:6808/115848 conn(0x7f22ac06e800 0x7f22ac070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.303+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== osd.6 v2:172.21.15.157:6808/115848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f22ac071280 con 0x7f22ac06e800 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.303+0000 7f22c4ff9700 10 client.5106.objecter ms_dispatch 0x7f22c8001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.303+0000 7f22c4ff9700 10 client.5106.objecter _finish_command 1 = system:0 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.303+0000 7f22c4ff9700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 4 2023-09-30T01:08:46.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.303+0000 7f22c4ff9700 15 client.5106.objecter _session_command_op_remove 6 1 2023-09-30T01:08:46.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.316+0000 7f22ceb57700 10 client.5106.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:46.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.316+0000 7f22ceb57700 15 client.5106.objecter _session_command_op_assign -1 2 2023-09-30T01:08:46.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.316+0000 7f22ceb57700 20 client.5106.objecter _get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.316+0000 7f22ceb57700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 client.5106.objecter _get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 15 client.5106.objecter _session_command_op_remove -1 2 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 client.5106.objecter get_session s=0x7f22ac06e150 osd=6 3 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 15 client.5106.objecter _session_command_op_assign 6 2 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 4 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 client.5106.objecter _send_command 2 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 --> v2:172.21.15.157:6808/115848 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22c80b0230 con 0x7f22ac06e800 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22c4ff9700 1 -- 172.21.15.115:0/2956883424 <== osd.6 v2:172.21.15.157:6808/115848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f22c80b0230 con 0x7f22ac06e800 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22c4ff9700 10 client.5106.objecter ms_dispatch 0x7f22c8001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:46.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22c4ff9700 10 client.5106.objecter _finish_command 2 = system:0 2023-09-30T01:08:46.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22c4ff9700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 4 2023-09-30T01:08:46.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22c4ff9700 15 client.5106.objecter _session_command_op_remove 6 2 2023-09-30T01:08:46.331 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:46.331 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 librados: watch_flush enter 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 librados: watch_flush exit 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 client.5106.objecter close_session for osd.6 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 >> v2:172.21.15.157:6808/115848 conn(0x7f22ac06e800 msgr2=0x7f22ac070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.157:6808/115848 conn(0x7f22ac06e800 0x7f22ac070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 client.5106.objecter put_session s=0x7f22ac06e150 osd=6 1 2023-09-30T01:08:46.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 client.5106.objecter shutdown clearing up homeless session... 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 client.5106.objecter successfully canceled tick 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 >> v2:172.21.15.115:6816/115635 conn(0x7f22ac068890 msgr2=0x7f22ac06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 --2- 172.21.15.115:0/2956883424 >> v2:172.21.15.115:6816/115635 conn(0x7f22ac068890 0x7f22ac06ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22b40093f0 tx=0x7f22b4017000 comp rx=0 tx=0).stop 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 10 monclient: shutdown 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 msgr2=0x7f22c80aacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.317+0000 7f22ceb57700 1 --2- 172.21.15.115:0/2956883424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f22c80a5640 0x7f22c80aacb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00c790 tx=0x7f22bc007e10 comp rx=0 tx=0).stop 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22cd0f4700 1 -- 172.21.15.115:0/2956883424 reap_dead start 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 shutdown_connections 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 >> 172.21.15.115:0/2956883424 conn(0x7f22c8068e40 msgr2=0x7f22c8138b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 shutdown_connections 2023-09-30T01:08:46.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22ceb57700 1 -- 172.21.15.115:0/2956883424 wait complete. 2023-09-30T01:08:46.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.318+0000 7f22ceb57700 1 librados: shutdown 2023-09-30T01:08:46.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T01:08:46.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.453+0000 7f9684f2c700 10 monclient: get_monmap_and_config 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.453+0000 7f9684f2c700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.453+0000 7f9684f2c700 10 monclient: monmap: 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:46.454633+0000 2023-09-30T01:08:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:46.454633+0000 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:46.455 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.454+0000 7f9684f2c700 1 Processor -- start 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.454+0000 7f9684f2c700 1 -- start start 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.454+0000 7f9684f2c700 10 monclient: init 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.454+0000 7f9684f2c700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.454+0000 7f9684f2c700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-b con 0x7f96800ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96800aecc0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:46.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-a con 0x7f96801415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96800a7040 con 0x7f96800aecc0 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96800a57f0 con 0x7f96801415c0 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800a5480 con 0x7f96800ae8f0 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient(hunting): authenticate will time out at 1387.905079s 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36212/0 (socket says 172.21.15.115:36212) 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57000/0 (socket says 172.21.15.115:57000) 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 -- 172.21.15.115:0/1581494352 learned_addr learned my addr 172.21.15.115:0/1581494352 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): get_auth_request con 0x7f96800ae8f0 auth_method 0 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): get_auth_request con 0x7f96801415c0 auth_method 0 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 -- 172.21.15.115:0/1581494352 >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 msgr2=0x7f96800adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 --2- 172.21.15.115:0/1581494352 >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 -- 172.21.15.115:0/1581494352 >> v1:172.21.15.157:6789/0 conn(0x7f96800aecc0 legacy=0x7f96800b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 monclient: found mon.noname-b 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 -- 172.21.15.115:0/1581494352 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800a55c0 con 0x7f96800ae8f0 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967ed9d700 1 --2- 172.21.15.115:0/1581494352 >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967e59c700 1 --2- 172.21.15.115:0/1581494352 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f96740099b0 tx=0x7f9674009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9df58dd650923113 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 1 -- 172.21.15.115:0/1581494352 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674009ea0 con 0x7f96800ae8f0 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: dump: 2023-09-30T01:08:46.474 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.475 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 5 monclient: authenticate success, global_id 4811 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 1 -- 172.21.15.115:0/1581494352 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967400b860 con 0x7f96800ae8f0 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:46.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f9684f2c700 4 set_mon_vals no callback set 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 1 -- 172.21.15.115:0/1581494352 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400bc00 con 0x7f96800ae8f0 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.455+0000 7f967d59a700 10 monclient: dump: 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 10 monclient: shutdown 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 -- 172.21.15.115:0/1581494352 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 msgr2=0x7f96800a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 --2- 172.21.15.115:0/1581494352 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f96740099b0 tx=0x7f9674009250 comp rx=0 tx=0).stop 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 -- 172.21.15.115:0/1581494352 shutdown_connections 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 --2- 172.21.15.115:0/1581494352 >> v2:172.21.15.115:3300/0 conn(0x7f96801415c0 0x7f96800adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 --2- 172.21.15.115:0/1581494352 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96800ae8f0 0x7f96800a5d10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 -- 172.21.15.115:0/1581494352 >> 172.21.15.115:0/1581494352 conn(0x7f968006a460 msgr2=0x7f96800630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 -- 172.21.15.115:0/1581494352 shutdown_connections 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 1 -- 172.21.15.115:0/1581494352 wait complete. 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.456+0000 7f9684f2c700 10 monclient: monmap: 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:46.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 librados: starting msgr at 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 librados: starting objecter 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 Processor -- start 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 -- start start 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 librados: setting wanted keys 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 librados: calling monclient init 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient: init 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:08:46.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96800ae8f0 0x7f968013f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-b con 0x7f96800ae8f0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 0x7f96801513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-a con 0x7f96800aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): picked mon.noname-c con 0x7f96801415c0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800ab2d0 con 0x7f96800aecc0 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96801518e0 con 0x7f96800ae8f0 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9680151aa0 con 0x7f96801415c0 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f9684f2c700 10 monclient(hunting): authenticate will time out at 1387.907928s 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.457+0000 7f967ed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 0x7f96801513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36214/0 (socket says 172.21.15.115:36214) 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 -- 172.21.15.115:0/3992071948 learned_addr learned my addr 172.21.15.115:0/3992071948 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): get_auth_request con 0x7f96801415c0 auth_method 0 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): get_auth_request con 0x7f96800aecc0 auth_method 0 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96800ae8f0 0x7f968013f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): get_auth_request con 0x7f96800ae8f0 auth_method 0 2023-09-30T01:08:46.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient(hunting): handle_auth_done global_id 4814 payload 824 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 -- 172.21.15.115:0/3992071948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96800ae8f0 msgr2=0x7f968013f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96800ae8f0 0x7f968013f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 -- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 msgr2=0x7f96801513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 0x7f96801513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 monclient: found mon.noname-c 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 -- 172.21.15.115:0/3992071948 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678009a20 con 0x7f96801415c0 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967dd9b700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 0x7f96801513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967ed9d700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9678009f20 tx=0x7f967800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 client.?.objecter ms_handle_connect 0x7f96801415c0 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800cb70 con 0x7f96801415c0 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: dump: 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.488 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 5 monclient: authenticate success, global_id 4814 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800f040 con 0x7f96801415c0 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 10 monclient: _renew_subs 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678014480 con 0x7f96801415c0 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96740094f0 con 0x7f96801415c0 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f967cd99700 4 set_mon_vals no callback set 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f966e7fc700 10 monclient: dump: 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.490 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 10 client.4814.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 10 monclient: _renew_subs 2023-09-30T01:08:46.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968013aae0 con 0x7f96801415c0 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.458+0000 7f9684f2c700 1 librados: init done 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.459+0000 7f9684f2c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.459+0000 7f9684f2c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.459+0000 7f9684f2c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.459+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800a8fe0 con 0x7f96801415c0 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9678003020 con 0x7f96801415c0 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 1 --2- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 0x7f966006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f967e59c700 1 --2- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 0x7f966006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9678085210 con 0x7f96801415c0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 10 client.4814.objecter ms_dispatch 0x7f9680001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 3 client.4814.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 3 client.4814.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f967e59c700 10 monclient: get_auth_request con 0x7f9660068890 auth_method 0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.460+0000 7f966e7fc700 20 client.4814.objecter dump_active .. 0 homeless 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.461+0000 7f966e7fc700 10 client.4814.objecter ms_handle_connect 0x7f9660068890 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.461+0000 7f967e59c700 1 --2- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 0x7f966006ad40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f96740099b0 tx=0x7f9674017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.464+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9678052ab0 con 0x7f96801415c0 2023-09-30T01:08:46.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.464+0000 7f966e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.464+0000 7f966e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:46.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.643+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f9680141a40 con 0x7f9660068890 2023-09-30T01:08:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.644+0000 7f966e7fc700 1 -- 172.21.15.115:0/3992071948 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9680141a40 con 0x7f9660068890 2023-09-30T01:08:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 10 librados: watch_flush enter 2023-09-30T01:08:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 10 librados: watch_flush exit 2023-09-30T01:08:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 20 client.4814.objecter shutdown clearing up homeless session... 2023-09-30T01:08:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 10 client.4814.objecter successfully canceled tick 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 msgr2=0x7f966006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 0x7f966006ad40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f96740099b0 tx=0x7f9674017000 comp rx=0 tx=0).stop 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 10 monclient: shutdown 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 msgr2=0x7f968013a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9678009f20 tx=0x7f967800b040 comp rx=0 tx=0).stop 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 shutdown_connections 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> v2:172.21.15.115:6816/115635 conn(0x7f9660068890 0x7f966006ad40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f96801415c0 0x7f968013a360 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96800ae8f0 0x7f968013f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 --2- 172.21.15.115:0/3992071948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96800aecc0 0x7f96801513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.645+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 >> 172.21.15.115:0/3992071948 conn(0x7f968006a460 msgr2=0x7f96800ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.646+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 shutdown_connections 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.646+0000 7f9684f2c700 1 -- 172.21.15.115:0/3992071948 wait complete. 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.646+0000 7f9684f2c700 1 librados: shutdown 2023-09-30T01:08:46.648 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T01:08:46.658 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T01:08:46.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:08:46.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.778+0000 7fa7ed796700 10 monclient: get_monmap_and_config 2023-09-30T01:08:46.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.778+0000 7fa7ed796700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.778+0000 7fa7ed796700 10 monclient: monmap: 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:46.779522+0000 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:46.779522+0000 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.779 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:46.780 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:46.780 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:46.780 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:46.781 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:46.781 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 Processor -- start 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 -- start start 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient: init 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 0x7fa7e80a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-b con 0x7fa7e80b0a20 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa7e80a57f0 0x7fa7e8141540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-c con 0x7fa7e80a57f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7e80a5c00 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80a7a00 con 0x7fa7e80a5c00 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e80b19d0 con 0x7fa7e80a57f0 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80a6b50 con 0x7fa7e80b0a20 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7ed796700 10 monclient(hunting): authenticate will time out at 1388.229747s 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e77fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa7e80a5c00 0x7fa7e8141a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52842/0 (socket says 172.21.15.115:52842) 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 0x7fa7e80a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e77fe700 1 -- 172.21.15.115:0/342929580 learned_addr learned my addr 172.21.15.115:0/342929580 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:46.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e6ffd700 10 monclient(hunting): get_auth_request con 0x7fa7e80b0a20 auth_method 0 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.779+0000 7fa7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e67fc700 1 --2- 172.21.15.115:0/342929580 >> v2:172.21.15.157:3300/0 conn(0x7fa7e80a57f0 0x7fa7e8141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e67fc700 10 monclient(hunting): get_auth_request con 0x7fa7e80a57f0 auth_method 0 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_done global_id 4817 payload 274 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 -- 172.21.15.115:0/342929580 >> v2:172.21.15.157:3300/0 conn(0x7fa7e80a57f0 msgr2=0x7fa7e8141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 --2- 172.21.15.115:0/342929580 >> v2:172.21.15.157:3300/0 conn(0x7fa7e80a57f0 0x7fa7e8141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 1 -- 172.21.15.115:0/342929580 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936021829 0 0) 0x7fa7e80a7a00 con 0x7fa7e80a5c00 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 -- 172.21.15.115:0/342929580 >> v1:172.21.15.115:6789/0 conn(0x7fa7e80a5c00 legacy=0x7fa7e8141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 monclient: found mon.noname-b 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 -- 172.21.15.115:0/342929580 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80a6dc0 con 0x7fa7e80b0a20 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:46.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e6ffd700 1 --2- 172.21.15.115:0/342929580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 0x7fa7e80a74c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0098e0 tx=0x7fa7dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21af0af1cf336428 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 1 -- 172.21.15.115:0/342929580 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc009e80 con 0x7fa7e80b0a20 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: dump: 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.795 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 1 -- 172.21.15.115:0/342929580 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7dc00b860 con 0x7fa7e80b0a20 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 5 monclient: authenticate success, global_id 4817 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 1 -- 172.21.15.115:0/342929580 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00bbe0 con 0x7fa7e80b0a20 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 4 set_mon_vals no callback set 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7e5ffb700 10 monclient: dump: 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.797 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 10 monclient: shutdown 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 1 -- 172.21.15.115:0/342929580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 msgr2=0x7fa7e80a74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.780+0000 7fa7ed796700 1 --2- 172.21.15.115:0/342929580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 0x7fa7e80a74c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0098e0 tx=0x7fa7dc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 -- 172.21.15.115:0/342929580 shutdown_connections 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 --2- 172.21.15.115:0/342929580 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80b0a20 0x7fa7e80a74c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 --2- 172.21.15.115:0/342929580 >> v2:172.21.15.157:3300/0 conn(0x7fa7e80a57f0 0x7fa7e8141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 -- 172.21.15.115:0/342929580 >> 172.21.15.115:0/342929580 conn(0x7fa7e806a210 msgr2=0x7fa7e80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 -- 172.21.15.115:0/342929580 shutdown_connections 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 1 -- 172.21.15.115:0/342929580 wait complete. 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.781+0000 7fa7ed796700 10 monclient: monmap: 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:46.799 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 librados: starting msgr at 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 librados: starting objecter 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 Processor -- start 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 -- start start 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 librados: setting wanted keys 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 librados: calling monclient init 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient: init 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:08:46.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 0x7fa7e80aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-c con 0x7fa7e80a57f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e80a5c00 0x7fa7e80a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-a con 0x7fa7e80a5c00 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e80b0a20 0x7fa7e813d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): picked mon.noname-b con 0x7fa7e80b0a20 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80ab140 con 0x7fa7e80a5c00 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e813f3e0 con 0x7fa7e80b0a20 2023-09-30T01:08:46.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80a8410 con 0x7fa7e80a57f0 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7ed796700 10 monclient(hunting): authenticate will time out at 1388.232789s 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 0x7fa7e80aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e80a5c00 0x7fa7e80a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 0x7fa7e80aa030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36234/0 (socket says 172.21.15.115:36234) 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 1 -- 172.21.15.115:0/2068538359 learned_addr learned my addr 172.21.15.115:0/2068538359 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 10 monclient(hunting): get_auth_request con 0x7fa7e80a57f0 auth_method 0 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): get_auth_request con 0x7fa7e80a5c00 auth_method 0 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e77fe700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e80b0a20 0x7fa7e813d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e77fe700 10 monclient(hunting): get_auth_request con 0x7fa7e80b0a20 auth_method 0 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.782+0000 7fa7e77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 10 monclient(hunting): handle_auth_done global_id 5118 payload 824 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 -- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 msgr2=0x7fa7e80aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 0x7fa7e80aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 -- 172.21.15.115:0/2068538359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e80b0a20 msgr2=0x7fa7e813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e80b0a20 0x7fa7e813d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 monclient: found mon.noname-a 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 -- 172.21.15.115:0/2068538359 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc009420 con 0x7fa7e80a5c00 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e6ffd700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fa7e80a57f0 0x7fa7e80aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e67fc700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e80a5c00 0x7fa7e80a9280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800d890 tx=0x7fa7d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 client.?.objecter ms_handle_connect 0x7fa7e80a5c00 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80125f0 con 0x7fa7e80a5c00 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: dump: 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.806 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d8012770 con 0x7fa7e80a5c00 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 5 monclient: authenticate success, global_id 5118 2023-09-30T01:08:46.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8012b30 con 0x7fa7e80a5c00 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ceffd700 10 monclient: dump: 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.808 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 10 monclient: _renew_subs 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e813dba0 con 0x7fa7e80a5c00 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7e57fa700 4 set_mon_vals no callback set 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 10 client.5118.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 10 monclient: _renew_subs 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e813e010 con 0x7fa7e80a5c00 2023-09-30T01:08:46.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.783+0000 7fa7ed796700 1 librados: init done 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 10 client.5118.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 15 client.5118.objecter _session_command_op_assign -1 1 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 20 client.5118.objecter _get_session osd=-1 returning homeless 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 10 client.5118.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7e8072960 con 0x7fa7e80a5c00 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.784+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa7d8013480 con 0x7fa7e80a5c00 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.115:6816/115635 conn(0x7fa7d40688c0 0x7fa7d406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa7d80841e0 con 0x7fa7e80a5c00 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 client.5118.objecter ms_dispatch 0x7fa7e8001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:46.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 3 client.5118.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 3 client.5118.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e6ffd700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.115:6816/115635 conn(0x7fa7d40688c0 0x7fa7d406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e6ffd700 10 monclient: get_auth_request con 0x7fa7d40688c0 auth_method 0 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 client.5118.objecter checking command 1 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.157:6812/115891 conn(0x7fa7d406e830 0x7fa7d4070c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter _get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 15 client.5118.objecter _session_command_op_remove -1 1 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter _get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 15 client.5118.objecter _session_command_op_assign 7 1 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 4 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 client.5118.objecter _send_command 1 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 --> v2:172.21.15.157:6812/115891 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7d40712b0 con 0x7fa7d406e830 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 20 client.5118.objecter dump_active .. 0 homeless 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d8086820 con 0x7fa7e80a5c00 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 client.5118.objecter ms_handle_connect 0x7fa7d40688c0 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e77fe700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.157:6812/115891 conn(0x7fa7d406e830 0x7fa7d4070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e6ffd700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.115:6816/115635 conn(0x7fa7d40688c0 0x7fa7d406ad70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0093a0 tx=0x7fa7dc017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e77fe700 10 monclient: get_auth_request con 0x7fa7d406e830 auth_method 0 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7ceffd700 10 client.5118.objecter ms_handle_connect 0x7fa7d406e830 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.785+0000 7fa7e77fe700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.157:6812/115891 conn(0x7fa7d406e830 0x7fa7d4070c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.786+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== osd.7 v2:172.21.15.157:6812/115891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa7d40712b0 con 0x7fa7d406e830 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.786+0000 7fa7ceffd700 10 client.5118.objecter ms_dispatch 0x7fa7e8001550 command_reply(tid 1: 0 ) v1 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.786+0000 7fa7ceffd700 10 client.5118.objecter _finish_command 1 = system:0 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.786+0000 7fa7ceffd700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 4 2023-09-30T01:08:46.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.786+0000 7fa7ceffd700 15 client.5118.objecter _session_command_op_remove 7 1 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 client.5118.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 15 client.5118.objecter _session_command_op_assign -1 2 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter _get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter _get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 15 client.5118.objecter _session_command_op_remove -1 2 2023-09-30T01:08:46.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter get_session s=0x7fa7d406e180 osd=7 3 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 15 client.5118.objecter _session_command_op_assign 7 2 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 4 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 client.5118.objecter _send_command 2 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 --> v2:172.21.15.157:6812/115891 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa7e80b0360 con 0x7fa7d406e830 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ceffd700 1 -- 172.21.15.115:0/2068538359 <== osd.7 v2:172.21.15.157:6812/115891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa7e80b0360 con 0x7fa7d406e830 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ceffd700 10 client.5118.objecter ms_dispatch 0x7fa7e8001550 command_reply(tid 2: 0 ) v1 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ceffd700 10 client.5118.objecter _finish_command 2 = system:0 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ceffd700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 4 2023-09-30T01:08:46.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ceffd700 15 client.5118.objecter _session_command_op_remove 7 2 2023-09-30T01:08:46.816 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T01:08:46.816 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:08:46.816 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T01:08:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 librados: watch_flush enter 2023-09-30T01:08:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 librados: watch_flush exit 2023-09-30T01:08:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 client.5118.objecter close_session for osd.7 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 >> v2:172.21.15.157:6812/115891 conn(0x7fa7d406e830 msgr2=0x7fa7d4070c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.157:6812/115891 conn(0x7fa7d406e830 0x7fa7d4070c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter put_session s=0x7fa7d406e180 osd=7 1 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 20 client.5118.objecter shutdown clearing up homeless session... 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 10 client.5118.objecter successfully canceled tick 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 >> v2:172.21.15.115:6816/115635 conn(0x7fa7d40688c0 msgr2=0x7fa7d406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.800+0000 7fa7ed796700 1 --2- 172.21.15.115:0/2068538359 >> v2:172.21.15.115:6816/115635 conn(0x7fa7d40688c0 0x7fa7d406ad70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0093a0 tx=0x7fa7dc017000 comp rx=0 tx=0).stop 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 10 monclient: shutdown 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e80a5c00 msgr2=0x7fa7e80a9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 --2- 172.21.15.115:0/2068538359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7e80a5c00 0x7fa7e80a9280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800d890 tx=0x7fa7d8008040 comp rx=0 tx=0).stop 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7e77fe700 1 -- 172.21.15.115:0/2068538359 reap_dead start 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 shutdown_connections 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 >> 172.21.15.115:0/2068538359 conn(0x7fa7e806a210 msgr2=0x7fa7e80b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 shutdown_connections 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 -- 172.21.15.115:0/2068538359 wait complete. 2023-09-30T01:08:46.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.801+0000 7fa7ed796700 1 librados: shutdown 2023-09-30T01:08:46.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 10 monclient: get_monmap_and_config 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 10 monclient: monmap: 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:46.937086+0000 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:46.937086+0000 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.936 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:46.937 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 1 Processor -- start 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 1 -- start start 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.936+0000 7fcdbb311700 10 monclient: init 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T01:08:46.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-b con 0x7fcdb40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcdb40aecc0 0x7fcdb40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-c con 0x7fcdb40aecc0 addr v2:172.21.15.157:3300/0 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-a con 0x7fcdb41415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb40a7040 con 0x7fcdb41415c0 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb40b19b0 con 0x7fcdb40aecc0 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40a57f0 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdbb311700 10 monclient(hunting): authenticate will time out at 1388.387597s 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36248/0 (socket says 172.21.15.115:36248) 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57016/0 (socket says 172.21.15.115:57016) 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 1 -- 172.21.15.115:0/434273020 learned_addr learned my addr 172.21.15.115:0/434273020 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): get_auth_request con 0x7fcdb40ae8f0 auth_method 0 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): get_auth_request con 0x7fcdb41415c0 auth_method 0 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb88ac700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.157:3300/0 conn(0x7fcdb40aecc0 0x7fcdb40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb88ac700 10 monclient(hunting): get_auth_request con 0x7fcdb40aecc0 auth_method 0 2023-09-30T01:08:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb88ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb88ac700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.937+0000 7fcdb88ac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_done global_id 4823 payload 274 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 -- 172.21.15.115:0/434273020 >> v2:172.21.15.157:3300/0 conn(0x7fcdb40aecc0 msgr2=0x7fcdb40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.157:3300/0 conn(0x7fcdb40aecc0 0x7fcdb40b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 -- 172.21.15.115:0/434273020 >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 msgr2=0x7fcdb40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 monclient: found mon.noname-b 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.115:3301/0 2023-09-30T01:08:46.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 -- 172.21.15.115:0/434273020 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb40a5480 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb98ae700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/434273020 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcdb406a8b0 tx=0x7fcda4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad83d86d1fdf15ae server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 1 -- 172.21.15.115:0/434273020 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4009e70 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: dump: 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.950 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 1 -- 172.21.15.115:0/434273020 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda400b860 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 5 monclient: authenticate success, global_id 4823 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:46.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 1 -- 172.21.15.115:0/434273020 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda400bc00 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 4 set_mon_vals no callback set 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdabfff700 10 monclient: dump: 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.952 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 10 monclient: shutdown 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 -- 172.21.15.115:0/434273020 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 msgr2=0x7fcdb40a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 --2- 172.21.15.115:0/434273020 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcdb406a8b0 tx=0x7fcda4009250 comp rx=0 tx=0).stop 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 -- 172.21.15.115:0/434273020 shutdown_connections 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.115:3300/0 conn(0x7fcdb41415c0 0x7fcdb40adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 --2- 172.21.15.115:0/434273020 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40a5d10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 --2- 172.21.15.115:0/434273020 >> v2:172.21.15.157:3300/0 conn(0x7fcdb40aecc0 0x7fcdb40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.938+0000 7fcdbb311700 1 -- 172.21.15.115:0/434273020 >> 172.21.15.115:0/434273020 conn(0x7fcdb406a460 msgr2=0x7fcdb40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 -- 172.21.15.115:0/434273020 shutdown_connections 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 -- 172.21.15.115:0/434273020 wait complete. 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 10 monclient: build_initial_monmap 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 10 monclient: monmap: 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:46.954 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 librados: starting msgr at 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 librados: starting objecter 2023-09-30T01:08:46.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 Processor -- start 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 -- start start 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 librados: setting wanted keys 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 1 librados: calling monclient init 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.939+0000 7fcdbb311700 10 monclient: init 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-c con 0x7fcdb40ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdb40aecc0 0x7fcdb4140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-b con 0x7fcdb40aecc0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): picked mon.noname-a con 0x7fcdb41415c0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40a9f90 con 0x7fcdb41415c0 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40ab340 con 0x7fcdb40aecc0 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb40ab5b0 con 0x7fcdb40ae8f0 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdbb311700 10 monclient(hunting): authenticate will time out at 1388.390700s 2023-09-30T01:08:46.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:36260/0 (socket says 172.21.15.115:36260) 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57032/0 (socket says 172.21.15.115:57032) 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 1 -- 172.21.15.115:0/3573580675 learned_addr learned my addr 172.21.15.115:0/3573580675 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): get_auth_request con 0x7fcdb40ae8f0 auth_method 0 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): get_auth_request con 0x7fcdb41415c0 auth_method 0 2023-09-30T01:08:46.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb88ac700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdb40aecc0 0x7fcdb4140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb90ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb88ac700 10 monclient(hunting): get_auth_request con 0x7fcdb40aecc0 auth_method 0 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb88ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:46.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb88ac700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.940+0000 7fcdb88ac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb88ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb88ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb88ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 10 monclient(hunting): handle_auth_done global_id 5124 payload 824 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 -- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 msgr2=0x7fcdb40aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 -- 172.21.15.115:0/3573580675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdb40aecc0 msgr2=0x7fcdb4140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdb40aecc0 0x7fcdb4140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 10 monclient: _finish_hunting 0 2023-09-30T01:08:46.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 monclient: found mon.noname-a 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 -- 172.21.15.115:0/3573580675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac009a20 con 0x7fcdb41415c0 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdb98ae700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009e70 tx=0x7fcdac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 client.?.objecter ms_handle_connect 0x7fcdb41415c0 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac00caa0 con 0x7fcdb41415c0 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: dump: 2023-09-30T01:08:46.961 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: _finish_auth 0 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:46.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdac00f040 con 0x7fcdb41415c0 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 5 monclient: authenticate success, global_id 5124 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 10 monclient: _renew_subs 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda40094f0 con 0x7fcdb41415c0 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac00ce70 con 0x7fcdb41415c0 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcda97fa700 10 monclient: dump: 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdab7fe700 4 set_mon_vals no callback set 2023-09-30T01:08:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 10 client.5124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 10 monclient: _renew_subs 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb40a9700 con 0x7fcdb41415c0 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.941+0000 7fcdbb311700 1 librados: init done 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.942+0000 7fcdbb311700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.942+0000 7fcdbb311700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.942+0000 7fcdbb311700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.942+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb40a9980 con 0x7fcdb41415c0 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.942+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcdac003020 con 0x7fcdb41415c0 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 1 --2- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 0x7fcd9806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcdac084590 con 0x7fcdb41415c0 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 10 client.5124.objecter ms_dispatch 0x7fcdb4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 3 client.5124.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 3 client.5124.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 0x7fcd9806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcdb90ad700 10 monclient: get_auth_request con 0x7fcd980688c0 auth_method 0 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 20 client.5124.objecter dump_active .. 0 homeless 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcda97fa700 10 client.5124.objecter ms_handle_connect 0x7fcd980688c0 2023-09-30T01:08:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.943+0000 7fcdb90ad700 1 --2- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 0x7fcd9806ad70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcda4003080 tx=0x7fcda4017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:46.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.947+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcdac051e50 con 0x7fcdb41415c0 2023-09-30T01:08:46.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.947+0000 7fcda97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:46.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:46.947+0000 7fcda97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:47.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.129+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdb40a9d60 con 0x7fcd980688c0 2023-09-30T01:08:47.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.129+0000 7fcda97fa700 1 -- 172.21.15.115:0/3573580675 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcdb40a9d60 con 0x7fcd980688c0 2023-09-30T01:08:47.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.130+0000 7fcdbb311700 10 librados: watch_flush enter 2023-09-30T01:08:47.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 10 librados: watch_flush exit 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 20 client.5124.objecter shutdown clearing up homeless session... 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 10 client.5124.objecter successfully canceled tick 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 msgr2=0x7fcd9806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 0x7fcd9806ad70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcda4003080 tx=0x7fcda4017000 comp rx=0 tx=0).stop 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 10 monclient: shutdown 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 msgr2=0x7fcdb4151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009e70 tx=0x7fcdac00b040 comp rx=0 tx=0).stop 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 shutdown_connections 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> v2:172.21.15.115:6816/115635 conn(0x7fcd980688c0 0x7fcd9806ad70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcdb41415c0 0x7fcdb4151110 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fcdb40ae8f0 0x7fcdb40aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 --2- 172.21.15.115:0/3573580675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdb40aecc0 0x7fcdb4140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 >> 172.21.15.115:0/3573580675 conn(0x7fcdb406a460 msgr2=0x7fcdb4140b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 shutdown_connections 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 -- 172.21.15.115:0/3573580675 wait complete. 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.131+0000 7fcdbb311700 1 librados: shutdown 2023-09-30T01:08:47.133 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T01:08:47.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.258+0000 7fc2188e5700 10 monclient: get_monmap_and_config 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.258+0000 7fc2188e5700 10 monclient: build_initial_monmap 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.258+0000 7fc2188e5700 10 monclient: monmap: 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:08:47.259730+0000 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:08:47.259730+0000 2023-09-30T01:08:47.259 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.259+0000 7fc2188e5700 1 Processor -- start 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.259+0000 7fc2188e5700 1 -- start start 2023-09-30T01:08:47.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.259+0000 7fc2188e5700 10 monclient: init 2023-09-30T01:08:47.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:47.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-09-30T01:08:47.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc2140ae8f0 0x7fc2140a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:47.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-a con 0x7fc2140ae8f0 addr v2:172.21.15.115:3300/0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2140aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2141415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140a7040 con 0x7fc2140aecc0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140a57f0 con 0x7fc2141415c0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140a5480 con 0x7fc2140ae8f0 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient(hunting): authenticate will time out at 1388.710141s 2023-09-30T01:08:47.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc2140ae8f0 0x7fc2140a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc211d9b700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc2140aecc0 0x7fc2140b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:52852/0 (socket says 172.21.15.115:52852) 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc211d9b700 1 -- 172.21.15.115:0/4141518550 learned_addr learned my addr 172.21.15.115:0/4141518550 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): get_auth_request con 0x7fc2140ae8f0 auth_method 0 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:47.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient(hunting): handle_auth_done global_id 5127 payload 274 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 -- 172.21.15.115:0/4141518550 >> v1:172.21.15.157:6789/0 conn(0x7fc2141415c0 legacy=0x7fc2140adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 -- 172.21.15.115:0/4141518550 >> v1:172.21.15.115:6789/0 conn(0x7fc2140aecc0 legacy=0x7fc2140b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 monclient: found mon.noname-a 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 -- 172.21.15.115:0/4141518550 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140a55c0 con 0x7fc2140ae8f0 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 1 -- 172.21.15.115:0/4141518550 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909183388 0 0) 0x7fc2140a7040 con 0x7fc2140aecc0 2023-09-30T01:08:47.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21259c700 1 --2- 172.21.15.115:0/4141518550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140a5d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098e0 tx=0x7fc2080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cef2886bedf6d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 1 -- 172.21.15.115:0/4141518550 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208009e80 con 0x7fc2140ae8f0 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: dump: 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.273 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: _finish_auth 0 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 5 monclient: authenticate success, global_id 5127 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:08:47.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 1 -- 172.21.15.115:0/4141518550 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20800b820 con 0x7fc2140ae8f0 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 10 monclient: get_monmap_and_config success 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc2188e5700 4 set_mon_vals no callback set 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 1 -- 172.21.15.115:0/4141518550 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20800bba0 con 0x7fc2140ae8f0 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.260+0000 7fc21159a700 10 monclient: dump: 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:47.275 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 10 monclient: shutdown 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:47.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 -- 172.21.15.115:0/4141518550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 msgr2=0x7fc2140a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 --2- 172.21.15.115:0/4141518550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140a5d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc2080098e0 tx=0x7fc2080091a0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 -- 172.21.15.115:0/4141518550 shutdown_connections 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 --2- 172.21.15.115:0/4141518550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140a5d10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 -- 172.21.15.115:0/4141518550 >> 172.21.15.115:0/4141518550 conn(0x7fc21406a460 msgr2=0x7fc2140630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 -- 172.21.15.115:0/4141518550 shutdown_connections 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 1 -- 172.21.15.115:0/4141518550 wait complete. 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 10 monclient: build_initial_monmap 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.261+0000 7fc2188e5700 10 monclient: monmap: 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:08:47.277 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 librados: starting msgr at 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 librados: starting objecter 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 Processor -- start 2023-09-30T01:08:47.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 -- start start 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 librados: setting wanted keys 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 librados: calling monclient init 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient: init 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient: _reopen_session rank -1 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-a con 0x7fc2140ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc2140aecc0 0x7fc21413e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-c con 0x7fc2140aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 0x7fc2141514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): picked mon.noname-b con 0x7fc2141415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:08:47.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a8c50 con 0x7fc2140ae8f0 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140aa9a0 con 0x7fc2141415c0 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a98a0 con 0x7fc2140aecc0 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): _renew_subs 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc2188e5700 10 monclient(hunting): authenticate will time out at 1388.712884s 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:57056/0 (socket says 172.21.15.115:57056) 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc211d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc2140aecc0 0x7fc21413e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:47.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 1 -- 172.21.15.115:0/702303487 learned_addr learned my addr 172.21.15.115:0/702303487 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 10 monclient(hunting): get_auth_request con 0x7fc2140ae8f0 auth_method 0 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.262+0000 7fc21259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): get_auth_request con 0x7fc2140aecc0 auth_method 0 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:47.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc211d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 0x7fc2141514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 10 monclient(hunting): get_auth_request con 0x7fc2141415c0 auth_method 0 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient(hunting): handle_auth_done global_id 5133 payload 824 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 -- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc2140aecc0 msgr2=0x7fc21413e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc2140aecc0 0x7fc21413e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 -- 172.21.15.115:0/702303487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 msgr2=0x7fc2141514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 0x7fc2141514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient: _finish_hunting 0 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 monclient: found mon.noname-a 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 -- 172.21.15.115:0/702303487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208009420 con 0x7fc2140ae8f0 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc212d9d700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 0x7fc2141514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc21259c700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc2140b0920 tx=0x7fc208016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 client.?.objecter ms_handle_connect 0x7fc2140ae8f0 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:08:47.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20800ba60 con 0x7fc2140ae8f0 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: dump: 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:47.284 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: _finish_auth 0 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208020070 con 0x7fc2140ae8f0 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 5 monclient: authenticate success, global_id 5133 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 10 monclient: _renew_subs 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2140a9ac0 con 0x7fc2140ae8f0 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20801b6a0 con 0x7fc2140ae8f0 2023-09-30T01:08:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2027fc700 10 monclient: dump: 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:08:47.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc210d99700 4 set_mon_vals no callback set 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 10 client.5133.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 10 monclient: _renew_subs 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc214151c40 con 0x7fc2140ae8f0 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.263+0000 7fc2188e5700 1 librados: init done 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.264+0000 7fc2188e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.264+0000 7fc2188e5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.264+0000 7fc2188e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.264+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2140aac80 con 0x7fc2140ae8f0 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.264+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc208022030 con 0x7fc2140ae8f0 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 1 --2- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 0x7fc1f406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:08:47.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc208085d30 con 0x7fc2140ae8f0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 10 client.5133.objecter ms_dispatch 0x7fc214001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 3 client.5133.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 3 client.5133.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc211d9b700 1 --2- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 0x7fc1f406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc211d9b700 10 monclient: get_auth_request con 0x7fc1f4068890 auth_method 0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 20 client.5133.objecter dump_active .. 0 homeless 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc2027fc700 10 client.5133.objecter ms_handle_connect 0x7fc1f4068890 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.265+0000 7fc211d9b700 1 --2- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 0x7fc1f406ad40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc214152550 tx=0x7fc204007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.269+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2080535f0 con 0x7fc2140ae8f0 2023-09-30T01:08:47.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.269+0000 7fc2027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:08:47.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.269+0000 7fc2027fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:08:47.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.449+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2140aaae0 con 0x7fc1f4068890 2023-09-30T01:08:47.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.452+0000 7fc2027fc700 1 -- 172.21.15.115:0/702303487 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72235 (secure 0 0 0) 0x7fc204001420 con 0x7fc1f4068890 2023-09-30T01:08:47.453 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 10 librados: watch_flush enter 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 10 librados: watch_flush exit 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 20 client.5133.objecter shutdown clearing up homeless session... 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 10 client.5133.objecter successfully canceled tick 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 msgr2=0x7fc1f406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 0x7fc1f406ad40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc214152550 tx=0x7fc204007d30 comp rx=0 tx=0).stop 2023-09-30T01:08:47.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 10 monclient: shutdown 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 msgr2=0x7fc2140aa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc2140b0920 tx=0x7fc208016000 comp rx=0 tx=0).stop 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 shutdown_connections 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> v2:172.21.15.115:6816/115635 conn(0x7fc1f4068890 0x7fc1f406ad40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2141415c0 0x7fc2141514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2140ae8f0 0x7fc2140aa2f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 --2- 172.21.15.115:0/702303487 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7fc2140aecc0 0x7fc21413e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:08:47.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 >> 172.21.15.115:0/702303487 conn(0x7fc21406a460 msgr2=0x7fc21413fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:08:47.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 shutdown_connections 2023-09-30T01:08:47.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 -- 172.21.15.115:0/702303487 wait complete. 2023-09-30T01:08:47.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:08:47.454+0000 7fc2188e5700 1 librados: shutdown 2023-09-30T01:08:47.457 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:08:47.466 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2023-09-30T01:08:46.066086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41924,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932796,"statfs":{"total":773094113280,"available":773051183104,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002474"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.035352+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:35.035352+0000","last_peered":"2023-09-30T01:03:35.035352+0000","last_clean":"2023-09-30T01:03:35.035352+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:35.035352+0000","last_undegraded":"2023-09-30T01:03:35.035352+0000","last_fullsized":"2023-09-30T01:03:35.035352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:47:04.097685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.034568+0000","last_change":"2023-09-30T01:03:35.034298+0000","last_active":"2023-09-30T01:03:35.034568+0000","last_peered":"2023-09-30T01:03:35.034568+0000","last_clean":"2023-09-30T01:03:35.034568+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:03:35.034568+0000","last_undegraded":"2023-09-30T01:03:35.034568+0000","last_fullsized":"2023-09-30T01:03:35.034568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:25:28.516404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033731+0000","last_change":"2023-09-30T01:03:35.032957+0000","last_active":"2023-09-30T01:03:35.033731+0000","last_peered":"2023-09-30T01:03:35.033731+0000","last_clean":"2023-09-30T01:03:35.033731+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:03:35.033731+0000","last_undegraded":"2023-09-30T01:03:35.033731+0000","last_fullsized":"2023-09-30T01:03:35.033731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:58.321312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033762+0000","last_change":"2023-09-30T01:03:35.033117+0000","last_active":"2023-09-30T01:03:35.033762+0000","last_peered":"2023-09-30T01:03:35.033762+0000","last_clean":"2023-09-30T01:03:35.033762+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:03:35.033762+0000","last_undegraded":"2023-09-30T01:03:35.033762+0000","last_fullsized":"2023-09-30T01:03:35.033762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.119259+0000","last_change":"2023-09-30T01:03:34.029437+0000","last_active":"2023-09-30T01:03:35.119259+0000","last_peered":"2023-09-30T01:03:35.119259+0000","last_clean":"2023-09-30T01:03:35.119259+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:03:35.119259+0000","last_undegraded":"2023-09-30T01:03:35.119259+0000","last_fullsized":"2023-09-30T01:03:35.119259+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:33.014324+0000","last_clean_scrub_stamp":"2023-09-30T01:03:33.014324+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:29:35.468649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121010+0000","last_change":"2023-09-30T01:03:35.120518+0000","last_active":"2023-09-30T01:03:35.121010+0000","last_peered":"2023-09-30T01:03:35.121010+0000","last_clean":"2023-09-30T01:03:35.121010+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:03:35.121010+0000","last_undegraded":"2023-09-30T01:03:35.121010+0000","last_fullsized":"2023-09-30T01:03:35.121010+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:33.948968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.033682+0000","last_change":"2023-09-30T01:03:35.033168+0000","last_active":"2023-09-30T01:03:35.033682+0000","last_peered":"2023-09-30T01:03:35.033682+0000","last_clean":"2023-09-30T01:03:35.033682+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:03:35.033682+0000","last_undegraded":"2023-09-30T01:03:35.033682+0000","last_fullsized":"2023-09-30T01:03:35.033682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:24.849061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.122330+0000","last_change":"2023-09-30T01:03:35.120836+0000","last_active":"2023-09-30T01:03:35.122330+0000","last_peered":"2023-09-30T01:03:35.122330+0000","last_clean":"2023-09-30T01:03:35.122330+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:03:35.122330+0000","last_undegraded":"2023-09-30T01:03:35.122330+0000","last_fullsized":"2023-09-30T01:03:35.122330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:34:31.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T01:03:35.121269+0000","last_change":"2023-09-30T01:03:35.120771+0000","last_active":"2023-09-30T01:03:35.121269+0000","last_peered":"2023-09-30T01:03:35.121269+0000","last_clean":"2023-09-30T01:03:35.121269+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:03:35.121269+0000","last_undegraded":"2023-09-30T01:03:35.121269+0000","last_fullsized":"2023-09-30T01:03:35.121269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:17.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.625,"15min":0.82399999999999995},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83199999999999996,"5min":6.883,"15min":6.883},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.69399999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70699999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.45100000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.88900000000000001},"min":{"1min":0.44400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77100000000000002,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.64600000000000002,"15min":0.83799999999999997},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.77700000000000002,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.63100000000000001,"15min":0.84099999999999997},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.378},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.98899999999999999},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.752,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.73599999999999999,"15min":1.038},"min":{"1min":0.48499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79100000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.495}]},{"osd":4,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.61599999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69499999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72199999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68100000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65400000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73099999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.441,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64700000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68799999999999994,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.65600000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47899999999999998,"15min":0.46500000000000002},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64600000000000002,"5min":1.103,"15min":1.103},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.48099999999999998},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":1.014,"15min":1.014},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Sep 30 01:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68300000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58799999999999997},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.70199999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5672,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366168,"statfs":{"total":96636764160,"available":96630956032,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52100000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64500000000000002,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.435}]},{"osd":2,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.373,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":1.012,"15min":1.012},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.42199999999999999,"15min":0.45300000000000001},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.60699999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.65200000000000002,"5min":1.218,"15min":1.218},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.67900000000000005,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.505}]},{"osd":4,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76300000000000001,"5min":1.333,"15min":1.333},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.44700000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80800000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60999999999999999,"15min":0.628},"min":{"1min":0.38100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.84299999999999997}]},{"osd":6,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.628,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.50700000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.873,"5min":1.04,"15min":1.04},"last":0.81999999999999995}]},{"osd":7,"last update":"Sat Sep 30 01:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.45100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.83499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.44900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90600000000000003,"5min":1.254,"15min":1.254},"last":0.85599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.224,"5min":0.193,"15min":0.193},"max":{"1min":0.53400000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.246},{"interface":"front","average":{"1min":0.374,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.54800000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.30199999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39000000000000001,"15min":0.39900000000000002},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38700000000000001,"15min":0.41099999999999998},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52200000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.49199999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.377},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.437,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51400000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.099,"15min":1.099},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79100000000000004,"5min":1.091,"15min":1.091},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":1.073,"15min":1.073},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.67500000000000004,"15min":0.70199999999999996},"min":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81799999999999995,"5min":1.194,"15min":1.194},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:07:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65100000000000002,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85299999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.68600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82499999999999996,"5min":1.163,"15min":1.163},"last":0.46200000000000002}]},{"osd":7,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63300000000000001,"15min":0.68300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80200000000000005,"5min":1.393,"15min":1.393},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45600000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.85499999999999998,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.57899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.377,"15min":0.371},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38800000000000001,"15min":0.38700000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67700000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63500000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.44600000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.44800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.81599999999999995,"5min":1.085,"15min":1.085},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.62},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.624,"15min":0.624},"min":{"1min":0.437,"5min":0.435,"15min":0.435},"max":{"1min":0.84799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.622,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89600000000000002,"5min":1.232,"15min":1.232},"last":0.48099999999999998}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.52900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38200000000000001,"15min":0.51600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":5.476,"15min":5.476},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.49199999999999999,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.40999999999999998,"15min":0.46800000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52200000000000002,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.47699999999999998,"15min":0.63700000000000001},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.315},{"interface":"front","average":{"1min":0.376,"5min":0.46700000000000003,"15min":0.628},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56000000000000005,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.26000000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63400000000000001,"15min":0.67000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80200000000000005,"5min":1.925,"15min":1.925},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.71999999999999997},"min":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83399999999999996,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.69799999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60499999999999998,"15min":0.67200000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80700000000000005,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.65500000000000003,"15min":0.76900000000000002},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67100000000000004,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66400000000000003,"15min":0.76800000000000002},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":5.649,"15min":5.649},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.65200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84899999999999998,"5min":1.855,"15min":1.855},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":1.895,"15min":1.895},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.64700000000000002,"15min":0.70299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.873,"5min":3.294,"15min":3.294},"last":0.71599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.749,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.64400000000000002,"15min":0.71999999999999997},"min":{"1min":0.373,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.72899999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67200000000000004,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.64600000000000002},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":1.387,"15min":1.387},"last":0.55200000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47199999999999998,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.433},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.38500000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.504,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.59899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69899999999999995,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Sep 30 01:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54800000000000004,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68899999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.5},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64700000000000002},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.56699999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52100000000000002,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.497},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.432,"5min":0.38,"15min":0.38},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72699999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.376},{"interface":"front","average":{"1min":0.505,"5min":0.498,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73299999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51200000000000001,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81899999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92100000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Sep 30 01:07:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.63300000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":1.014,"15min":1.014},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T01:03:33.014324+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.471 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.472 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.472 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.472 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:08:47.472 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T01:09:07.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.602+0000 7f27bb945700 10 monclient: get_monmap_and_config 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.602+0000 7f27bb945700 10 monclient: build_initial_monmap 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.602+0000 7f27bb945700 10 monclient: monmap: 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:09:07.603487+0000 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:09:07.603487+0000 2023-09-30T01:09:07.603 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.603+0000 7f27bb945700 1 Processor -- start 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.603+0000 7f27bb945700 1 -- start start 2023-09-30T01:09:07.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.603+0000 7f27bb945700 10 monclient: init 2023-09-30T01:09:07.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:07.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f27b40ae8f0 0x7f27b40a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-c con 0x7f27b40ae8f0 addr v2:172.21.15.157:3300/0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27b40aecc0 addr v1:172.21.15.115:6789/0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27b41415c0 addr v1:172.21.15.157:6789/0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b40a7040 con 0x7f27b40aecc0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b40a57f0 con 0x7f27b41415c0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b40a5480 con 0x7f27b40ae8f0 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 10 monclient(hunting): authenticate will time out at 1409.054109s 2023-09-30T01:09:07.618 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b8ee0700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f27b40aecc0 0x7f27b40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:49470/0 (socket says 172.21.15.115:49470) 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b8ee0700 1 -- 172.21.15.115:0/2310658078 learned_addr learned my addr 172.21.15.115:0/2310658078 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 1 --2- 172.21.15.115:0/2310658078 >> v2:172.21.15.157:3300/0 conn(0x7f27b40ae8f0 0x7f27b40a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): get_auth_request con 0x7f27b40ae8f0 auth_method 0 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738287644 0 0) 0x7f27b40a7040 con 0x7f27b40aecc0 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): my global_id is 5136 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0003570 con 0x7f27b40aecc0 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36281733 0 0) 0x7f27b40a57f0 con 0x7f27b41415c0 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): my global_id is 4888 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b40a7040 con 0x7f27b41415c0 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973379783 0 0) 0x7f27a0003570 con 0x7f27b40aecc0 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 >> v2:172.21.15.157:3300/0 conn(0x7f27b40ae8f0 msgr2=0x7f27b40a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 --2- 172.21.15.115:0/2310658078 >> v2:172.21.15.157:3300/0 conn(0x7f27b40ae8f0 0x7f27b40a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 >> v1:172.21.15.157:6789/0 conn(0x7f27b41415c0 legacy=0x7f27b40adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.115:6789/0 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b40a55c0 con 0x7f27b40aecc0 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: _finish_auth 0 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3661308906 0 0) 0x7f27a4001470 con 0x7f27b40aecc0 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 5 monclient: authenticate success, global_id 5136 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:09:07.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: dump: 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] but i'm connected to v1:172.21.15.115:6789/0, reconnecting 2023-09-30T01:09:07.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 >> v1:172.21.15.115:6789/0 conn(0x7f27b40aecc0 legacy=0x7f27b40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 0x7f27a0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): picked mon.a con 0x7f27a0003bf0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27a0004810 0x7f27a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): picked mon.c con 0x7f27a0004810 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a00051c0 0x7f27a000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): picked mon.b con 0x7f27a00051c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40a5480 con 0x7f27a0003bf0 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a000c0b0 con 0x7f27a00051c0 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a000c270 con 0x7f27a0004810 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27abfff700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27bb945700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b96e1700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 0x7f27a0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.604+0000 7f27b8ee0700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27a0004810 0x7f27a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): get_auth_request con 0x7f27a0003bf0 auth_method 0 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:09:07.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): get_auth_request con 0x7f27a0004810 auth_method 0 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a00051c0 0x7f27a000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): get_auth_request con 0x7f27a00051c0 auth_method 0 2023-09-30T01:09:07.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient(hunting): handle_auth_done global_id 5136 payload 293 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 -- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27a0004810 msgr2=0x7f27a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27a0004810 0x7f27a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 -- 172.21.15.115:0/2310658078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a00051c0 msgr2=0x7f27a000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a00051c0 0x7f27a000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient: _finish_hunting 0 2023-09-30T01:09:07.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 monclient: found mon.a 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 -- 172.21.15.115:0/2310658078 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0003570 con 0x7f27a0003bf0 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 10 monclient: _finish_auth 0 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b9ee2700 1 -- 172.21.15.115:0/2310658078 reap_dead start 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b8ee0700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27a0004810 0x7f27a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27b96e1700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 0x7f27a0003fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009c00 tx=0x7f27b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd2ed65047723aa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b000e710 con 0x7f27a0003bf0 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:09:07.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: dump: 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b000e870 con 0x7f27a0003bf0 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:07.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 1 -- 172.21.15.115:0/2310658078 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b000ec30 con 0x7f27a0003bf0 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27abfff700 10 monclient: dump: 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27bb945700 10 monclient: get_monmap_and_config success 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.605+0000 7f27bb945700 4 set_mon_vals no callback set 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 10 monclient: shutdown 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 -- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 msgr2=0x7f27a0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 0x7f27a0003fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009c00 tx=0x7f27b00091a0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 -- 172.21.15.115:0/2310658078 shutdown_connections 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 --2- 172.21.15.115:0/2310658078 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27a0003bf0 0x7f27a0003fe0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 -- 172.21.15.115:0/2310658078 >> 172.21.15.115:0/2310658078 conn(0x7f27b406a460 msgr2=0x7f27b40630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 -- 172.21.15.115:0/2310658078 shutdown_connections 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 1 -- 172.21.15.115:0/2310658078 wait complete. 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 10 monclient: build_initial_monmap 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.606+0000 7f27bb945700 10 monclient: monmap: 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:09:07.631 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 librados: starting msgr at 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 librados: starting objecter 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 Processor -- start 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 -- start start 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 librados: setting wanted keys 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 librados: calling monclient init 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient: init 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a0003bf0 0x7f27b413d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-b con 0x7f27a0003bf0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:09:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-a con 0x7f27b40a8890 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 0x7f27b40a9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): picked mon.noname-c con 0x7f27b414eb00 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40a7d00 con 0x7f27b40a8890 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b414de50 con 0x7f27a0003bf0 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b414e0a0 con 0x7f27b414eb00 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.607+0000 7f27bb945700 10 monclient(hunting): authenticate will time out at 1409.057989s 2023-09-30T01:09:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:37858/0 (socket says 172.21.15.115:37858) 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 0x7f27b40a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 -- 172.21.15.115:0/833917116 learned_addr learned my addr 172.21.15.115:0/833917116 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): get_auth_request con 0x7f27b40a8890 auth_method 0 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): get_auth_request con 0x7f27b414eb00 auth_method 0 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a0003bf0 0x7f27b413d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): get_auth_request con 0x7f27a0003bf0 auth_method 0 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b96e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient(hunting): handle_auth_done global_id 5139 payload 824 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 -- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 msgr2=0x7f27b40a9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 0x7f27b40a9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 -- 172.21.15.115:0/833917116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a0003bf0 msgr2=0x7f27b413d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a0003bf0 0x7f27b413d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient: _finish_hunting 0 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 monclient: found mon.noname-a 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 -- 172.21.15.115:0/833917116 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b40a9580 con 0x7f27b40a8890 2023-09-30T01:09:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b9ee2700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 0x7f27b40a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27b8ee0700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f27a4000e10 tx=0x7f27a4001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 client.?.objecter ms_handle_connect 0x7f27b40a8890 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4018030 con 0x7f27b40a8890 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: dump: 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.637 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: _finish_auth 0 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a4007740 con 0x7f27b40a8890 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 5 monclient: authenticate success, global_id 5139 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 10 monclient: _renew_subs 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:09:07.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b40a9770 con 0x7f27b40a8890 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a400fae0 con 0x7f27b40a8890 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27a97fa700 10 monclient: dump: 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:07.639 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27ab7fe700 4 set_mon_vals no callback set 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 10 client.5139.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 10 monclient: _renew_subs 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:09:07.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b41516d0 con 0x7f27b40a8890 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.608+0000 7f27bb945700 1 librados: init done 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.609+0000 7f27bb945700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.609+0000 7f27bb945700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.609+0000 7f27bb945700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.115:3300/0 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.609+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b4072960 con 0x7f27b40a8890 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f27a401b070 con 0x7f27b40a8890 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 1 --2- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 0x7f279806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27b96e1700 1 --2- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 0x7f279806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f27a4019070 con 0x7f27b40a8890 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 10 client.5139.objecter ms_dispatch 0x7f27b4001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 3 client.5139.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:09:07.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27b96e1700 10 monclient: get_auth_request con 0x7f2798068890 auth_method 0 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 3 client.5139.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 20 client.5139.objecter dump_active .. 0 homeless 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27a97fa700 10 client.5139.objecter ms_handle_connect 0x7f2798068890 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.610+0000 7f27b96e1700 1 --2- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 0x7f279806ad40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f27b00091e0 tx=0x7f27b001a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.614+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27a4051640 con 0x7f27b40a8890 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.614+0000 7f27a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:07.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.614+0000 7f27a97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:09:07.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.790+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27b414e260 con 0x7f2798068890 2023-09-30T01:09:07.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.792+0000 7f27a97fa700 1 -- 172.21.15.115:0/833917116 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72777 (secure 0 0 0) 0x7f27b414e260 con 0x7f2798068890 2023-09-30T01:09:07.793 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:09:07.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.794+0000 7f27bb945700 10 librados: watch_flush enter 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 10 librados: watch_flush exit 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 20 client.5139.objecter shutdown clearing up homeless session... 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 10 client.5139.objecter successfully canceled tick 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 msgr2=0x7f279806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 0x7f279806ad40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f27b00091e0 tx=0x7f27b001a000 comp rx=0 tx=0).stop 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 10 monclient: shutdown 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 msgr2=0x7f27b414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f27a4000e10 tx=0x7f27a4001e10 comp rx=0 tx=0).stop 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 shutdown_connections 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> v2:172.21.15.115:6816/115635 conn(0x7f2798068890 0x7f279806ad40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7f27b414eb00 0x7f27b40a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a0003bf0 0x7f27b413d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 --2- 172.21.15.115:0/833917116 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27b40a8890 0x7f27b414e5c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 >> 172.21.15.115:0/833917116 conn(0x7f27b406a460 msgr2=0x7f27b40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 shutdown_connections 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 -- 172.21.15.115:0/833917116 wait complete. 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:07.795+0000 7f27bb945700 1 librados: shutdown 2023-09-30T01:09:07.797 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:09:07.806 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2023-09-30T01:09:06.070451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41896,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932824,"statfs":{"total":773094113280,"available":773051211776,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:03:36.283059+0000","last_change":"2023-09-30T01:03:35.034283+0000","last_active":"2023-09-30T01:03:36.283059+0000","last_peered":"2023-09-30T01:03:36.283059+0000","last_clean":"2023-09-30T01:03:36.283059+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:03:36.283059+0000","last_undegraded":"2023-09-30T01:03:36.283059+0000","last_fullsized":"2023-09-30T01:03:36.283059+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:03:34.019800+0000","last_clean_scrub_stamp":"2023-09-30T01:03:34.019800+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.196724+0000","last_change":"2023-09-30T01:08:52.196698+0000","last_active":"2023-09-30T01:08:52.196724+0000","last_peered":"2023-09-30T01:08:52.196724+0000","last_clean":"2023-09-30T01:08:52.196724+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:08:52.196724+0000","last_undegraded":"2023-09-30T01:08:52.196724+0000","last_fullsized":"2023-09-30T01:08:52.196724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.196587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.196587+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.196587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:07:33.955764+0000","scrub_duration":0.001922619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:09:02.703003+0000","last_change":"2023-09-30T01:09:02.702978+0000","last_active":"2023-09-30T01:09:02.703003+0000","last_peered":"2023-09-30T01:09:02.703003+0000","last_clean":"2023-09-30T01:09:02.703003+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:09:02.703003+0000","last_undegraded":"2023-09-30T01:09:02.703003+0000","last_fullsized":"2023-09-30T01:09:02.703003+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:09:02.702876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:09:02.702876+0000","last_clean_scrub_stamp":"2023-09-30T01:09:02.702876+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:52.521640+0000","scrub_duration":0.001854847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.659314+0000","last_change":"2023-09-30T01:08:52.659279+0000","last_active":"2023-09-30T01:08:52.659314+0000","last_peered":"2023-09-30T01:08:52.659314+0000","last_clean":"2023-09-30T01:08:52.659314+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:08:52.659314+0000","last_undegraded":"2023-09-30T01:08:52.659314+0000","last_fullsized":"2023-09-30T01:08:52.659314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.659142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.659142+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.659142+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:59.893980+0000","scrub_duration":0.0021008200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'119","reported_seq":173,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:45.436656+0000","last_change":"2023-09-30T01:08:45.436636+0000","last_active":"2023-09-30T01:08:45.436656+0000","last_peered":"2023-09-30T01:08:45.436656+0000","last_clean":"2023-09-30T01:08:45.436656+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:08:45.436656+0000","last_undegraded":"2023-09-30T01:08:45.436656+0000","last_fullsized":"2023-09-30T01:08:45.436656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'119","last_scrub_stamp":"2023-09-30T01:08:45.436560+0000","last_deep_scrub":"17'119","last_deep_scrub_stamp":"2023-09-30T01:08:45.436560+0000","last_clean_scrub_stamp":"2023-09-30T01:08:45.436560+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:29:01.894931+0000","scrub_duration":0.0046049560000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:48.575191+0000","last_change":"2023-09-30T01:08:48.575165+0000","last_active":"2023-09-30T01:08:48.575191+0000","last_peered":"2023-09-30T01:08:48.575191+0000","last_clean":"2023-09-30T01:08:48.575191+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:08:48.575191+0000","last_undegraded":"2023-09-30T01:08:48.575191+0000","last_fullsized":"2023-09-30T01:08:48.575191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:48.575063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:48.575063+0000","last_clean_scrub_stamp":"2023-09-30T01:08:48.575063+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:03:57.949524+0000","scrub_duration":0.0015626889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:49.589919+0000","last_change":"2023-09-30T01:08:49.589885+0000","last_active":"2023-09-30T01:08:49.589919+0000","last_peered":"2023-09-30T01:08:49.589919+0000","last_clean":"2023-09-30T01:08:49.589919+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:08:49.589919+0000","last_undegraded":"2023-09-30T01:08:49.589919+0000","last_fullsized":"2023-09-30T01:08:49.589919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:49.589743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:49.589743+0000","last_clean_scrub_stamp":"2023-09-30T01:08:49.589743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:13:10.590933+0000","scrub_duration":0.001730381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:59.648361+0000","last_change":"2023-09-30T01:08:59.648331+0000","last_active":"2023-09-30T01:08:59.648361+0000","last_peered":"2023-09-30T01:08:59.648361+0000","last_clean":"2023-09-30T01:08:59.648361+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:08:59.648361+0000","last_undegraded":"2023-09-30T01:08:59.648361+0000","last_fullsized":"2023-09-30T01:08:59.648361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:59.648191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:59.648191+0000","last_clean_scrub_stamp":"2023-09-30T01:08:59.648191+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:40:06.750631+0000","scrub_duration":0.00189865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.072113+0000","last_change":"2023-09-30T01:08:52.072088+0000","last_active":"2023-09-30T01:08:52.072113+0000","last_peered":"2023-09-30T01:08:52.072113+0000","last_clean":"2023-09-30T01:08:52.072113+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:08:52.072113+0000","last_undegraded":"2023-09-30T01:08:52.072113+0000","last_fullsized":"2023-09-30T01:08:52.072113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.071995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.071995+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.071995+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:53:50.853019+0000","scrub_duration":0.001550367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.625,"15min":0.82399999999999995},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83199999999999996,"5min":6.883,"15min":6.883},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.69399999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70699999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.88900000000000001},"min":{"1min":0.44400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77100000000000002,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.64600000000000002,"15min":0.83799999999999997},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.77700000000000002,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.63100000000000001,"15min":0.84099999999999997},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.98899999999999999},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.752,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.73599999999999999,"15min":1.038},"min":{"1min":0.48499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79100000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.67700000000000005}]},{"osd":4,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.61599999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69499999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72199999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68100000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54700000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53600000000000003,"15min":0.52700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53200000000000003,"15min":0.51200000000000001},"min":{"1min":0.40600000000000003,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69199999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69599999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72499999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.627},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.46600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65500000000000003,"5min":1.103,"15min":1.103},"last":0.76500000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.61299999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.72999999999999998,"5min":1.014,"15min":1.014},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69399999999999995,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.75700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35899999999999999,"15min":0.40200000000000002},"min":{"1min":0.23699999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52600000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.38600000000000001,"15min":0.40699999999999997},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53600000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36499999999999999,"15min":0.38100000000000001},"min":{"1min":0.22800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47199999999999998,"5min":1.012,"15min":1.012},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40699999999999997,"15min":0.44800000000000001},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56100000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.55000000000000004,"5min":1.218,"15min":1.218},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41599999999999998,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.505,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.64800000000000002},"min":{"1min":0.46200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77700000000000002,"5min":1.333,"15min":1.333},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.433,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57299999999999995,"15min":0.60499999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81299999999999994,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.628},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.84299999999999997,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.78600000000000003}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88900000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83799999999999997,"5min":1.04,"15min":1.04},"last":0.61699999999999999}]},{"osd":7,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.82499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.67100000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85599999999999998,"5min":1.254,"15min":1.254},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.224,"5min":0.193,"15min":0.193},"max":{"1min":0.53400000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.54800000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39000000000000001,"15min":0.39900000000000002},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38700000000000001,"15min":0.41099999999999998},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52200000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.49199999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.372},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.437,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51400000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.438,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73799999999999999,"5min":1.099,"15min":1.099},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":1.091,"15min":1.091},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":1.073,"15min":1.073},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.67500000000000004,"15min":0.70199999999999996},"min":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81799999999999995,"5min":1.194,"15min":1.194},"last":0.505}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.63200000000000001,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77900000000000003,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63800000000000001,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":1.163,"15min":1.163},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63300000000000001,"15min":0.68300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80200000000000005,"5min":1.393,"15min":1.393},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45600000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.85499999999999998,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.58599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.37},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50800000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.40999999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.38900000000000001},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55400000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88300000000000001,"5min":1.085,"15min":1.085},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61299999999999999,"15min":0.63100000000000001},"min":{"1min":0.46700000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80600000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.749},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77300000000000002,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81999999999999995,"5min":1.232,"15min":1.232},"last":0.70099999999999996}]},{"osd":7,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.436,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.76400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38200000000000001,"15min":0.51600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":5.476,"15min":5.476},"last":0.309},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.49199999999999999,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.40999999999999998,"15min":0.46800000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52200000000000002,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.47699999999999998,"15min":0.63700000000000001},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.46700000000000003,"15min":0.628},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56000000000000005,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63400000000000001,"15min":0.67000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80200000000000005,"5min":1.925,"15min":1.925},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.71999999999999997},"min":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83399999999999996,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60499999999999998,"15min":0.67200000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80700000000000005,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.65500000000000003,"15min":0.76900000000000002},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67100000000000004,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66400000000000003,"15min":0.76800000000000002},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":5.649,"15min":5.649},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.65200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84899999999999998,"5min":1.855,"15min":1.855},"last":0.505}]},{"osd":7,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":1.895,"15min":1.895},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.64700000000000002,"15min":0.70299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.873,"5min":3.294,"15min":3.294},"last":0.60799999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.63800000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82099999999999995,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.39700000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.68400000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.70699999999999996},"min":{"1min":0.44600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.71699999999999997},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77700000000000002,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.57799999999999996},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58599999999999997,"15min":0.63900000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80900000000000005,"5min":1.387,"15min":1.387},"last":0.56799999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46300000000000002,"15min":0.495},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49099999999999999,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.52800000000000002},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54300000000000004,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.876,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.45200000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53700000000000003,"15min":0.57199999999999995},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.61599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72299999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5}]},{"osd":2,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66800000000000004,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69499999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76900000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.47099999999999997,"15min":0.497},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73199999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48599999999999999,"15min":0.505},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.76200000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38300000000000001}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.49399999999999999,"15min":0.52700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.749,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54100000000000004,"15min":0.56599999999999995},"min":{"1min":0.33800000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.78900000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.35599999999999998}]},{"osd":7,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.55100000000000005,"15min":0.58299999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80400000000000005,"5min":1.073,"15min":1.073},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82899999999999996,"5min":1.014,"15min":1.014},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:09:07.809 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T01:03:34.019800+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=3, tm_sec=34, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=8, tm_sec=42, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:09:07.809 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T01:09:27.810 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:09:27.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.940+0000 7ff872f4d700 10 monclient: get_monmap_and_config 2023-09-30T01:09:27.940 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.940+0000 7ff872f4d700 10 monclient: build_initial_monmap 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.940+0000 7ff872f4d700 10 monclient: monmap: 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:09:27.941607+0000 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:09:27.941607+0000 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:09:27.941 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:09:27.942 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:09:27.942 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:09:27.942 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:09:27.942 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.941+0000 7ff872f4d700 1 Processor -- start 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.941+0000 7ff872f4d700 1 -- start start 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.941+0000 7ff872f4d700 10 monclient: init 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff86c0ae8f0 addr v1:172.21.15.115:6789/0 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-b con 0x7ff86c0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff86c1415c0 0x7ff86c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-a con 0x7ff86c1415c0 addr v2:172.21.15.115:3300/0 2023-09-30T01:09:27.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c0a7040 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86c0a57f0 con 0x7ff86c1415c0 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0a5480 con 0x7ff86c0aecc0 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient(hunting): authenticate will time out at 1429.392171s 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7ff86c1415c0 0x7ff86c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44644/0 (socket says 172.21.15.115:44644) 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 1 -- 172.21.15.115:0/614014337 learned_addr learned my addr 172.21.15.115:0/614014337 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:09:27.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): get_auth_request con 0x7ff86c0aecc0 auth_method 0 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): get_auth_request con 0x7ff86c1415c0 auth_method 0 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:27.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient(hunting): handle_auth_done global_id 5142 payload 274 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 -- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 msgr2=0x7ff86c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 -- 172.21.15.115:0/614014337 >> v1:172.21.15.115:6789/0 conn(0x7ff86c0ae8f0 legacy=0x7ff86c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient: _finish_hunting 0 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 monclient: found mon.noname-a 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 -- 172.21.15.115:0/614014337 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c0a55c0 con 0x7ff86c1415c0 2023-09-30T01:09:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 1 -- 172.21.15.115:0/614014337 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748998396 0 0) 0x7ff86c0a7040 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86bfff700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff8714ea700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c1415c0 0x7ff86c0adb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff864009e60 tx=0x7ff864009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e1b219fb64261 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 1 -- 172.21.15.115:0/614014337 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86400e070 con 0x7ff86c1415c0 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: dump: 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.955 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 5 monclient: authenticate success, global_id 5142 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 1 -- 172.21.15.115:0/614014337 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86400bb60 con 0x7ff86c1415c0 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:27.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 10 monclient: get_monmap_and_config success 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 1 -- 172.21.15.115:0/614014337 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864012430 con 0x7ff86c1415c0 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff872f4d700 4 set_mon_vals no callback set 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.942+0000 7ff86b7fe700 10 monclient: dump: 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:27.957 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 10 monclient: shutdown 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 -- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c1415c0 msgr2=0x7ff86c0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c1415c0 0x7ff86c0adb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff864009e60 tx=0x7ff864009780 comp rx=0 tx=0).stop 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 -- 172.21.15.115:0/614014337 shutdown_connections 2023-09-30T01:09:27.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c1415c0 0x7ff86c0adb60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 --2- 172.21.15.115:0/614014337 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0aecc0 0x7ff86c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 -- 172.21.15.115:0/614014337 >> 172.21.15.115:0/614014337 conn(0x7ff86c06a460 msgr2=0x7ff86c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 -- 172.21.15.115:0/614014337 shutdown_connections 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.943+0000 7ff872f4d700 1 -- 172.21.15.115:0/614014337 wait complete. 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 10 monclient: build_initial_monmap 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 10 monclient: monmap: 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:27.959 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 librados: starting msgr at 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 librados: starting objecter 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 Processor -- start 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 -- start start 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 librados: setting wanted keys 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 1 librados: calling monclient init 2023-09-30T01:09:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.944+0000 7ff872f4d700 10 monclient: init 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-c con 0x7ff86c0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-a con 0x7ff86c0aecc0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86c1415c0 0x7ff86c14dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): picked mon.noname-b con 0x7ff86c1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0a8480 con 0x7ff86c0aecc0 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0a8170 con 0x7ff86c1415c0 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0a9c30 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff872f4d700 10 monclient(hunting): authenticate will time out at 1429.395344s 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:46560/0 (socket says 172.21.15.115:46560) 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44648/0 (socket says 172.21.15.115:44648) 2023-09-30T01:09:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 1 -- 172.21.15.115:0/3547680486 learned_addr learned my addr 172.21.15.115:0/3547680486 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): get_auth_request con 0x7ff86c0aecc0 auth_method 0 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): get_auth_request con 0x7ff86c0ae8f0 auth_method 0 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 -- 172.21.15.115:0/3547680486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86c1415c0 msgr2=0x7ff86c14dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86c1415c0 0x7ff86c14dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 -- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 msgr2=0x7ff86c0a7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient: _finish_hunting 0 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 monclient: found mon.noname-c 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:09:27.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 -- 172.21.15.115:0/3547680486 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580094f0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff86bfff700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff870ce9700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff8600011e0 tx=0x7ff860007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff868ff9700 10 client.?.objecter ms_handle_connect 0x7ff86c0ae8f0 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.945+0000 7ff868ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860011590 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: dump: 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:27.965 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: _finish_auth 0 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 5 monclient: authenticate success, global_id 4838 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 monclient: _renew_subs 2023-09-30T01:09:27.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8600116f0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff864009a20 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860011ab0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff86affd700 4 set_mon_vals no callback set 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff868ff9700 10 monclient: dump: 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.967 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 client.4838.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 monclient: _renew_subs 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c14e160 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 1 librados: init done 2023-09-30T01:09:27.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.946+0000 7ff872f4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.947+0000 7ff872f4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.947+0000 7ff872f4d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.947+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c0b18d0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.947+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff86001b3b0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.947+0000 7ff868ff9700 1 --2- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 0x7ff85006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff86bfff700 1 --2- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 0x7ff85006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff86bfff700 10 monclient: get_auth_request con 0x7ff850068890 auth_method 0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff8600833f0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 10 client.4838.objecter ms_dispatch 0x7ff86c001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 3 client.4838.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:09:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 3 client.4838.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 20 client.4838.objecter dump_active .. 0 homeless 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff868ff9700 10 client.4838.objecter ms_handle_connect 0x7ff850068890 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.948+0000 7ff86bfff700 1 --2- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 0x7ff85006ad40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff86c13e6e0 tx=0x7ff858009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.952+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff860050cb0 con 0x7ff86c0ae8f0 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.952+0000 7ff868ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:27.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:27.952+0000 7ff868ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:09:28.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.133+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 --> v2:172.21.15.115:6816/115635 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff86c072960 con 0x7ff850068890 2023-09-30T01:09:28.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.135+0000 7ff868ff9700 1 -- 172.21.15.115:0/3547680486 <== mgr.4103 v2:172.21.15.115:6816/115635 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72779 (secure 0 0 0) 0x7ff86c072960 con 0x7ff850068890 2023-09-30T01:09:28.136 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 10 librados: watch_flush enter 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 10 librados: watch_flush exit 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 20 client.4838.objecter shutdown clearing up homeless session... 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 10 client.4838.objecter successfully canceled tick 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 msgr2=0x7ff85006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 0x7ff85006ad40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff86c13e6e0 tx=0x7ff858009250 comp rx=0 tx=0).stop 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 10 monclient: shutdown 2023-09-30T01:09:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 msgr2=0x7ff86c0b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff8600011e0 tx=0x7ff860007d30 comp rx=0 tx=0).stop 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 shutdown_connections 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> v2:172.21.15.115:6816/115635 conn(0x7ff850068890 0x7ff85006ad40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86c1415c0 0x7ff86c14dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7ff86c0ae8f0 0x7ff86c0b4610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 --2- 172.21.15.115:0/3547680486 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff86c0aecc0 0x7ff86c0a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 >> 172.21.15.115:0/3547680486 conn(0x7ff86c06a460 msgr2=0x7ff86c13a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 shutdown_connections 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 -- 172.21.15.115:0/3547680486 wait complete. 2023-09-30T01:09:28.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.137+0000 7ff872f4d700 1 librados: shutdown 2023-09-30T01:09:28.140 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T01:09:28.149 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2023-09-30T01:09:28.075167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41896,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754932824,"statfs":{"total":773094113280,"available":773051211776,"internally_reserved":0,"allocated":1286144,"data_stored":1425680,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002754"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:09:04.664374+0000","last_change":"2023-09-30T01:09:04.664348+0000","last_active":"2023-09-30T01:09:04.664374+0000","last_peered":"2023-09-30T01:09:04.664374+0000","last_clean":"2023-09-30T01:09:04.664374+0000","last_became_active":"2023-09-30T01:03:35.034024+0000","last_became_peered":"2023-09-30T01:03:35.034024+0000","last_unstale":"2023-09-30T01:09:04.664374+0000","last_undegraded":"2023-09-30T01:09:04.664374+0000","last_fullsized":"2023-09-30T01:09:04.664374+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:09:04.664248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:09:04.664248+0000","last_clean_scrub_stamp":"2023-09-30T01:09:04.664248+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:37:07.449036+0000","scrub_duration":0.0015754149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.196724+0000","last_change":"2023-09-30T01:08:52.196698+0000","last_active":"2023-09-30T01:08:52.196724+0000","last_peered":"2023-09-30T01:08:52.196724+0000","last_clean":"2023-09-30T01:08:52.196724+0000","last_became_active":"2023-09-30T01:03:35.034063+0000","last_became_peered":"2023-09-30T01:03:35.034063+0000","last_unstale":"2023-09-30T01:08:52.196724+0000","last_undegraded":"2023-09-30T01:08:52.196724+0000","last_fullsized":"2023-09-30T01:08:52.196724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.196587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.196587+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.196587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:07:33.955764+0000","scrub_duration":0.001922619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:09:02.703003+0000","last_change":"2023-09-30T01:09:02.702978+0000","last_active":"2023-09-30T01:09:02.703003+0000","last_peered":"2023-09-30T01:09:02.703003+0000","last_clean":"2023-09-30T01:09:02.703003+0000","last_became_active":"2023-09-30T01:03:35.032703+0000","last_became_peered":"2023-09-30T01:03:35.032703+0000","last_unstale":"2023-09-30T01:09:02.703003+0000","last_undegraded":"2023-09-30T01:09:02.703003+0000","last_fullsized":"2023-09-30T01:09:02.703003+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:09:02.702876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:09:02.702876+0000","last_clean_scrub_stamp":"2023-09-30T01:09:02.702876+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:52.521640+0000","scrub_duration":0.001854847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.659314+0000","last_change":"2023-09-30T01:08:52.659279+0000","last_active":"2023-09-30T01:08:52.659314+0000","last_peered":"2023-09-30T01:08:52.659314+0000","last_clean":"2023-09-30T01:08:52.659314+0000","last_became_active":"2023-09-30T01:03:35.032893+0000","last_became_peered":"2023-09-30T01:03:35.032893+0000","last_unstale":"2023-09-30T01:08:52.659314+0000","last_undegraded":"2023-09-30T01:08:52.659314+0000","last_fullsized":"2023-09-30T01:08:52.659314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.659142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.659142+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.659142+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:08:59.893980+0000","scrub_duration":0.0021008200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'119","reported_seq":173,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:45.436656+0000","last_change":"2023-09-30T01:08:45.436636+0000","last_active":"2023-09-30T01:08:45.436656+0000","last_peered":"2023-09-30T01:08:45.436656+0000","last_clean":"2023-09-30T01:08:45.436656+0000","last_became_active":"2023-09-30T01:03:34.028555+0000","last_became_peered":"2023-09-30T01:03:34.028555+0000","last_unstale":"2023-09-30T01:08:45.436656+0000","last_undegraded":"2023-09-30T01:08:45.436656+0000","last_fullsized":"2023-09-30T01:08:45.436656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'119","last_scrub_stamp":"2023-09-30T01:08:45.436560+0000","last_deep_scrub":"17'119","last_deep_scrub_stamp":"2023-09-30T01:08:45.436560+0000","last_clean_scrub_stamp":"2023-09-30T01:08:45.436560+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:29:01.894931+0000","scrub_duration":0.0046049560000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:48.575191+0000","last_change":"2023-09-30T01:08:48.575165+0000","last_active":"2023-09-30T01:08:48.575191+0000","last_peered":"2023-09-30T01:08:48.575191+0000","last_clean":"2023-09-30T01:08:48.575191+0000","last_became_active":"2023-09-30T01:03:35.120222+0000","last_became_peered":"2023-09-30T01:03:35.120222+0000","last_unstale":"2023-09-30T01:08:48.575191+0000","last_undegraded":"2023-09-30T01:08:48.575191+0000","last_fullsized":"2023-09-30T01:08:48.575191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:48.575063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:48.575063+0000","last_clean_scrub_stamp":"2023-09-30T01:08:48.575063+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:03:57.949524+0000","scrub_duration":0.0015626889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:49.589919+0000","last_change":"2023-09-30T01:08:49.589885+0000","last_active":"2023-09-30T01:08:49.589919+0000","last_peered":"2023-09-30T01:08:49.589919+0000","last_clean":"2023-09-30T01:08:49.589919+0000","last_became_active":"2023-09-30T01:03:35.032828+0000","last_became_peered":"2023-09-30T01:03:35.032828+0000","last_unstale":"2023-09-30T01:08:49.589919+0000","last_undegraded":"2023-09-30T01:08:49.589919+0000","last_fullsized":"2023-09-30T01:08:49.589919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:49.589743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:49.589743+0000","last_clean_scrub_stamp":"2023-09-30T01:08:49.589743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:13:10.590933+0000","scrub_duration":0.001730381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:59.648361+0000","last_change":"2023-09-30T01:08:59.648331+0000","last_active":"2023-09-30T01:08:59.648361+0000","last_peered":"2023-09-30T01:08:59.648361+0000","last_clean":"2023-09-30T01:08:59.648361+0000","last_became_active":"2023-09-30T01:03:35.120578+0000","last_became_peered":"2023-09-30T01:03:35.120578+0000","last_unstale":"2023-09-30T01:08:59.648361+0000","last_undegraded":"2023-09-30T01:08:59.648361+0000","last_fullsized":"2023-09-30T01:08:59.648361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:59.648191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:59.648191+0000","last_clean_scrub_stamp":"2023-09-30T01:08:59.648191+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:40:06.750631+0000","scrub_duration":0.00189865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T01:08:52.072113+0000","last_change":"2023-09-30T01:08:52.072088+0000","last_active":"2023-09-30T01:08:52.072113+0000","last_peered":"2023-09-30T01:08:52.072113+0000","last_clean":"2023-09-30T01:08:52.072113+0000","last_became_active":"2023-09-30T01:03:35.120475+0000","last_became_peered":"2023-09-30T01:03:35.120475+0000","last_unstale":"2023-09-30T01:08:52.072113+0000","last_undegraded":"2023-09-30T01:08:52.072113+0000","last_fullsized":"2023-09-30T01:08:52.072113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:08:52.071995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:08:52.071995+0000","last_clean_scrub_stamp":"2023-09-30T01:08:52.071995+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:53:50.853019+0000","scrub_duration":0.001550367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10882,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.625,"15min":0.82399999999999995},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83199999999999996,"5min":6.883,"15min":6.883},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57699999999999996,"15min":0.69399999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70699999999999996,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.68000000000000005,"15min":0.88900000000000001},"min":{"1min":0.44400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77100000000000002,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.64600000000000002,"15min":0.83799999999999997},"min":{"1min":0.47899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.77700000000000002,"5min":6.8109999999999999,"15min":6.8109999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.63100000000000001,"15min":0.84099999999999997},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":6.9720000000000004,"15min":6.9720000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.98899999999999999},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.752,"5min":9.8979999999999997,"15min":9.8979999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.73599999999999999,"15min":1.038},"min":{"1min":0.48499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79100000000000004,"5min":9.9339999999999993,"15min":9.9339999999999993},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.61599999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69499999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72199999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42799999999999999}]},{"osd":6,"last update":"Sat Sep 30 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68100000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53600000000000003,"15min":0.52700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53200000000000003,"15min":0.51200000000000001},"min":{"1min":0.40600000000000003,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69199999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70399999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63800000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.503}]},{"osd":3,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69599999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.61399999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.46600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65500000000000003,"5min":1.103,"15min":1.103},"last":0.73299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.61299999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.72999999999999998,"5min":1.014,"15min":1.014},"last":0.66100000000000003}]},{"osd":7,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69399999999999995,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35899999999999999,"15min":0.40200000000000002},"min":{"1min":0.23699999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52600000000000002,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.38600000000000001,"15min":0.40699999999999997},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53600000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36499999999999999,"15min":0.38100000000000001},"min":{"1min":0.22800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47199999999999998,"5min":1.012,"15min":1.012},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40699999999999997,"15min":0.44800000000000001},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.56100000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.376}]},{"osd":3,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.55000000000000004,"5min":1.218,"15min":1.218},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41599999999999998,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.505,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.64800000000000002},"min":{"1min":0.46200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77700000000000002,"5min":1.333,"15min":1.333},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.64400000000000002},"min":{"1min":0.433,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57299999999999995,"15min":0.60499999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81299999999999994,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.84299999999999997,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.625}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88900000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83799999999999997,"5min":1.04,"15min":1.04},"last":0.69699999999999995}]},{"osd":7,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63,"15min":0.65000000000000002},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.82499999999999996,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.67100000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85599999999999998,"5min":1.254,"15min":1.254},"last":0.72999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.224,"5min":0.193,"15min":0.193},"max":{"1min":0.53400000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.54800000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.41099999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39000000000000001,"15min":0.39900000000000002},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53700000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38700000000000001,"15min":0.41099999999999998},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52200000000000002,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36799999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.49199999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.437,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51400000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.438,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73799999999999999,"5min":1.099,"15min":1.099},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82899999999999996,"5min":1.091,"15min":1.091},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65900000000000003,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84299999999999997,"5min":1.073,"15min":1.073},"last":0.625},{"interface":"front","average":{"1min":0.628,"5min":0.67500000000000004,"15min":0.70199999999999996},"min":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.81799999999999995,"5min":1.194,"15min":1.194},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.63200000000000001,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77900000000000003,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63800000000000001,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":1.163,"15min":1.163},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Sep 30 01:08:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63300000000000001,"15min":0.68300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80200000000000005,"5min":1.393,"15min":1.393},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.69599999999999995},"min":{"1min":0.45600000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.85499999999999998,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.37},"min":{"1min":0.247,"5min":0.217,"15min":0.217},"max":{"1min":0.48599999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50800000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.5}]},{"osd":1,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.437},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.38900000000000001},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.44,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55400000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88300000000000001,"5min":1.085,"15min":1.085},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61299999999999999,"15min":0.63100000000000001},"min":{"1min":0.46700000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77300000000000002,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.77800000000000002}]},{"osd":6,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81999999999999995,"5min":1.232,"15min":1.232},"last":0.81299999999999994}]},{"osd":7,"last update":"Sat Sep 30 01:08:49 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.436,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574921,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38200000000000001,"15min":0.51600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":5.476,"15min":5.476},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.49199999999999999,"5min":5.2789999999999999,"15min":5.2789999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.40999999999999998,"15min":0.46800000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52200000000000002,"5min":3.2469999999999999,"15min":3.2469999999999999},"last":0.443},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.47699999999999998,"15min":0.63700000000000001},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":7.4630000000000001,"15min":7.4630000000000001},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.46700000000000003,"15min":0.628},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56000000000000005,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63400000000000001,"15min":0.67000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80200000000000005,"5min":1.925,"15min":1.925},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.71999999999999997},"min":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83399999999999996,"5min":3.3610000000000002,"15min":3.3610000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60499999999999998,"15min":0.67200000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80700000000000005,"5min":3.3900000000000001,"15min":3.3900000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.65500000000000003,"15min":0.76900000000000002},"min":{"1min":0.43099999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67100000000000004,"5min":5.7229999999999999,"15min":5.7229999999999999},"last":0.5}]},{"osd":6,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66400000000000003,"15min":0.76800000000000002},"min":{"1min":0.45600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":5.649,"15min":5.649},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.65200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84899999999999998,"5min":1.855,"15min":1.855},"last":0.63200000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:08:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":1.895,"15min":1.895},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.64700000000000002,"15min":0.70299999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.873,"5min":3.294,"15min":3.294},"last":0.51900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73299999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.63800000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82099999999999995,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.39700000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56100000000000005,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.70699999999999996},"min":{"1min":0.44600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80000000000000004,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.71699999999999997},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77700000000000002,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.57799999999999996},"min":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58599999999999997,"15min":0.63900000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80900000000000005,"5min":1.387,"15min":1.387},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46300000000000002,"15min":0.495},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49099999999999999,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.29399999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.52800000000000002},"min":{"1min":0.34000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66200000000000003,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.442},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54300000000000004,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.876,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.38400000000000001}]},{"osd":7,"last update":"Sat Sep 30 01:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53700000000000003,"15min":0.57199999999999995},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.50700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":574906,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55500000000000005}]},{"osd":1,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72299999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66800000000000004,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69499999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76900000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45300000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.47099999999999997,"15min":0.497},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73199999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48599999999999999,"15min":0.505},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.76200000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.60399999999999998}]},{"osd":6,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.49399999999999999,"15min":0.52700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.749,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54100000000000004,"15min":0.56599999999999995},"min":{"1min":0.33800000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.78900000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.501}]},{"osd":7,"last update":"Sat Sep 30 01:08:44 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.55100000000000005,"15min":0.58299999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80400000000000005,"5min":1.073,"15min":1.073},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82899999999999996,"5min":1.014,"15min":1.014},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5441,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:09:28.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.267+0000 7efc00ea8700 10 monclient: get_monmap_and_config 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.267+0000 7efc00ea8700 10 monclient: build_initial_monmap 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.267+0000 7efc00ea8700 10 monclient: monmap: 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:09:28.268631+0000 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:09:28.268631+0000 2023-09-30T01:09:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.115:6789/0 mon.noname-a-legacy 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.157:6789/0 mon.noname-c-legacy 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-b 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:3: v2:172.21.15.115:3300/0 mon.noname-a 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:4: v2:172.21.15.157:3300/0 mon.noname-c 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 1 Processor -- start 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 1 -- start start 2023-09-30T01:09:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient: init 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-b con 0x7efbfc0ae8f0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7efbfc0aecc0 0x7efbfc0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-a con 0x7efbfc0aecc0 addr v2:172.21.15.115:3300/0 2023-09-30T01:09:28.278 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efbfc1415c0 addr v1:172.21.15.115:6789/0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.268+0000 7efc00ea8700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc0a7040 con 0x7efbfc1415c0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0b19b0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0a57f0 con 0x7efbfc0ae8f0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 10 monclient(hunting): authenticate will time out at 1429.719022s 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7efbfc0aecc0 0x7efbfc0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44664/0 (socket says 172.21.15.115:44664) 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 1 -- 172.21.15.115:0/3029362143 learned_addr learned my addr 172.21.15.115:0/3029362143 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:09:28.279 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): get_auth_request con 0x7efbfc0ae8f0 auth_method 0 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): get_auth_request con 0x7efbfc0aecc0 auth_method 0 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:28.280 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_done global_id 5151 payload 274 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 -- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 msgr2=0x7efbfc0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 -- 172.21.15.115:0/3029362143 >> v1:172.21.15.115:6789/0 conn(0x7efbfc1415c0 legacy=0x7efbfc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 monclient: found mon.noname-a 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.115:3300/0 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 1 -- 172.21.15.115:0/3029362143 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032436504 0 0) 0x7efbfc0a7040 con 0x7efbfc1415c0 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 -- 172.21.15.115:0/3029362143 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0a5480 con 0x7efbfc0aecc0 2023-09-30T01:09:28.281 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbfa59c700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf9d9b700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0aecc0 0x7efbfc0b1470 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098e0 tx=0x7efbf00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0c34a84479e211 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 1 -- 172.21.15.115:0/3029362143 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000b640 con 0x7efbfc0aecc0 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: dump: 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:28.282 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: _finish_auth 0 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 1 -- 172.21.15.115:0/3029362143 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000b7a0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 5 monclient: authenticate success, global_id 5151 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 1 -- 172.21.15.115:0/3029362143 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000bb40 con 0x7efbfc0aecc0 2023-09-30T01:09:28.283 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efbf959a700 10 monclient: dump: 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:28.284 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 10 monclient: get_monmap_and_config success 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.269+0000 7efc00ea8700 4 set_mon_vals no callback set 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 10 monclient: shutdown 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 -- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0aecc0 msgr2=0x7efbfc0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0aecc0 0x7efbfc0b1470 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098e0 tx=0x7efbf00091a0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 -- 172.21.15.115:0/3029362143 shutdown_connections 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0ae8f0 0x7efbfc0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 --2- 172.21.15.115:0/3029362143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0aecc0 0x7efbfc0b1470 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 -- 172.21.15.115:0/3029362143 >> 172.21.15.115:0/3029362143 conn(0x7efbfc06a460 msgr2=0x7efbfc0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 -- 172.21.15.115:0/3029362143 shutdown_connections 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 1 -- 172.21.15.115:0/3029362143 wait complete. 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 10 monclient: build_initial_monmap 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.270+0000 7efc00ea8700 10 monclient: monmap: 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.286 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.noname-a 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.noname-b 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.noname-c 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 librados: starting msgr at 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 librados: starting objecter 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 Processor -- start 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 -- start start 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 librados: setting wanted keys 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 librados: calling monclient init 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient: init 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient: _reopen_session rank -1 2023-09-30T01:09:28.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-a con 0x7efbfc0ae8f0 addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-c con 0x7efbfc0aecc0 addr [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbfc1415c0 0x7efbfc0abe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): picked mon.noname-b con 0x7efbfc1415c0 addr [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0ab790 con 0x7efbfc0ae8f0 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0a8440 con 0x7efbfc1415c0 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 1 -- --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0a8030 con 0x7efbfc0aecc0 2023-09-30T01:09:28.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): _renew_subs 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efc00ea8700 10 monclient(hunting): authenticate will time out at 1429.721863s 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbfa59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 1 --2- >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3301/0 says I am v2:172.21.15.115:44676/0 (socket says 172.21.15.115:44676) 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 1 -- 172.21.15.115:0/1707878238 learned_addr learned my addr 172.21.15.115:0/1707878238 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 10 monclient(hunting): get_auth_request con 0x7efbfc0aecc0 auth_method 0 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbfa59c700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:46586/0 (socket says 172.21.15.115:46586) 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbf9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.271+0000 7efbfa59c700 10 monclient(hunting): get_auth_request con 0x7efbfc0ae8f0 auth_method 0 2023-09-30T01:09:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient(hunting): handle_auth_done global_id 4844 payload 824 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfad9d700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbfc1415c0 0x7efbfc0abe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 -- 172.21.15.115:0/1707878238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbfc1415c0 msgr2=0x7efbfc0abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbfc1415c0 0x7efbfc0abe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 -- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 msgr2=0x7efbfc13e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 monclient: found mon.noname-c 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.115:3301/0 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 -- 172.21.15.115:0/1707878238 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec0094f0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbfa59c700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf9d9b700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0b1dd0 tx=0x7efbf0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 client.?.objecter ms_handle_connect 0x7efbfc0aecc0 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T01:09:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0000d10 con 0x7efbfc0aecc0 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: dump: 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:28.292 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: _finish_auth 0 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 5 monclient: authenticate success, global_id 4844 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000bb20 con 0x7efbfc0aecc0 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 10 monclient: _renew_subs 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0009420 con 0x7efbfc0aecc0 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0011650 con 0x7efbfc0aecc0 2023-09-30T01:09:28.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbea7fc700 10 monclient: dump: 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c6b635fb-787f-4de6-b95e-9004242e56e2 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:created 2023-09-30T01:02:46.397883+0000 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 17 (quincy) 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T01:09:28.294 INFO:teuthology.orchestra.run.smithi115.stderr:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efbf8d99700 4 set_mon_vals no callback set 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 10 client.4844.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 10 monclient: _renew_subs 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc0ac520 con 0x7efbfc0aecc0 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.272+0000 7efc00ea8700 1 librados: init done 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.273+0000 7efc00ea8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.273+0000 7efc00ea8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:28.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.273+0000 7efc00ea8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.273+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc0abc50 con 0x7efbfc0aecc0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.273+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efbf00119c0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 1 --2- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 0x7efbdc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbfa59c700 1 --2- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 0x7efbdc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efbf00851f0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 10 client.4844.objecter ms_dispatch 0x7efbfc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbfa59c700 10 monclient: get_auth_request con 0x7efbdc068890 auth_method 0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 3 client.4844.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 3 client.4844.objecter handle_osd_map decoding full epoch 17 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 20 client.4844.objecter dump_active .. 0 homeless 2023-09-30T01:09:28.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbea7fc700 10 client.4844.objecter ms_handle_connect 0x7efbdc068890 2023-09-30T01:09:28.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.274+0000 7efbfa59c700 1 --2- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 0x7efbdc06ad40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efbec006d20 tx=0x7efbec009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:09:28.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.278+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efbf0052b30 con 0x7efbfc0aecc0 2023-09-30T01:09:28.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.278+0000 7efbea7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:09:28.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.278+0000 7efbea7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:09:28.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.459+0000 7efc00ea8700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:09:28.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.459+0000 7efc00ea8700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:09:28.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.459+0000 7efc00ea8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.115:3301/0 2023-09-30T01:09:28.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.459+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 --> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7efbfc0ab900 con 0x7efbfc0aecc0 2023-09-30T01:09:28.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.463+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efbf00092a0 con 0x7efbfc0aecc0 2023-09-30T01:09:28.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.463+0000 7efbea7fc700 10 monclient: handle_config config(1 keys) v1 2023-09-30T01:09:28.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.463+0000 7efbf8d99700 4 set_mon_vals no callback set 2023-09-30T01:09:28.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.463+0000 7efbf8d99700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T01:09:28.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.464+0000 7efbea7fc700 1 -- 172.21.15.115:0/1707878238 <== mon.2 v2:172.21.15.115:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7efbf0056150 con 0x7efbfc0aecc0 2023-09-30T01:09:28.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.464+0000 7efbea7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:09:28.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.464+0000 7efbea7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:09:28.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 10 librados: watch_flush enter 2023-09-30T01:09:28.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 10 librados: watch_flush exit 2023-09-30T01:09:28.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 20 client.4844.objecter shutdown clearing up homeless session... 2023-09-30T01:09:28.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 10 client.4844.objecter successfully canceled tick 2023-09-30T01:09:28.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 msgr2=0x7efbdc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 0x7efbdc06ad40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efbec006d20 tx=0x7efbec009250 comp rx=0 tx=0).stop 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 10 monclient: shutdown 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 msgr2=0x7efbfc0a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0b1dd0 tx=0x7efbf0017000 comp rx=0 tx=0).stop 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 shutdown_connections 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> v2:172.21.15.115:6816/115635 conn(0x7efbdc068890 0x7efbdc06ad40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbfc1415c0 0x7efbfc0abe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efbfc0ae8f0 0x7efbfc13e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 --2- 172.21.15.115:0/1707878238 >> [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] conn(0x7efbfc0aecc0 0x7efbfc0a7af0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:09:28.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 >> 172.21.15.115:0/1707878238 conn(0x7efbfc06a460 msgr2=0x7efbfc0adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:09:28.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 shutdown_connections 2023-09-30T01:09:28.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 -- 172.21.15.115:0/1707878238 wait complete. 2023-09-30T01:09:28.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T01:09:28.466+0000 7efc00ea8700 1 librados: shutdown 2023-09-30T01:09:28.479 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:09:28.480 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T01:09:28.480 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T01:09:28.481 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T01:09:28.481 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.508 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T01:09:28.508 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T01:09:28.508 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.573 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T01:09:28.573 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T01:09:28.573 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.641 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T01:09:28.641 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T01:09:28.641 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.705 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T01:09:28.705 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T01:09:28.705 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.747 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T01:09:28.748 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T01:09:28.748 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.813 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T01:09:28.813 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T01:09:28.813 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.885 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T01:09:28.885 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T01:09:28.885 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.951 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T01:09:28.951 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T01:09:28.951 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T01:09:28.951 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:28.974 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T01:09:28.975 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T01:09:28.975 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:29.017 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T01:09:29.017 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T01:09:29.017 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T01:09:29.017 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:29.027 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T01:09:29.028 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T01:09:29.028 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:29.078 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T01:09:29.078 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T01:09:29.078 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:09:29.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-30T01:09:29.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-30T01:09:29.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T01:09:29.092 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T01:09:29.092 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T01:09:29.092 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2023-09-30T01:09:29.149 INFO:teuthology.orchestra.run.smithi115.stdout:1696035808.0253656 mgr.y (mgr.4103) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:09:29.149 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T01:09:29.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2023-09-30T01:09:29.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2023-09-30T01:09:29.290 INFO:teuthology.orchestra.run.smithi115.stdout:1696035808.0253656 mgr.y (mgr.4103) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:09:29.290 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:09:29.290 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T01:09:29.528 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:09:29.528 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T01:09:29.618 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:09:29.618 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T01:09:29.751 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:09:29.751 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T01:09:29.834 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:09:29.834 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T01:09:30.008 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:09:30.008 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T01:09:30.088 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:09:30.089 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T01:09:30.197 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:09:30.198 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T01:09:30.272 INFO:tasks.ceph:Archiving mon data... 2023-09-30T01:09:30.274 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/data/mon.a.tgz 2023-09-30T01:09:30.274 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T01:09:30.294 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.09nFdv5rvL 2023-09-30T01:09:30.294 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.09nFdv5rvL 2023-09-30T01:09:30.414 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.09nFdv5rvL 2023-09-30T01:09:30.501 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.09nFdv5rvL is 259KB 2023-09-30T01:09:30.595 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.09nFdv5rvL 2023-09-30T01:09:30.613 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/data/mon.c.tgz 2023-09-30T01:09:30.613 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T01:09:30.669 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.Jl8xu9T6Ha 2023-09-30T01:09:30.670 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Jl8xu9T6Ha 2023-09-30T01:09:30.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.Jl8xu9T6Ha 2023-09-30T01:09:30.865 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.Jl8xu9T6Ha is 259KB 2023-09-30T01:09:30.962 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.Jl8xu9T6Ha 2023-09-30T01:09:30.980 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/data/mon.b.tgz 2023-09-30T01:09:30.980 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T01:09:31.002 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.v6F2QIlHhn 2023-09-30T01:09:31.003 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.v6F2QIlHhn 2023-09-30T01:09:31.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.v6F2QIlHhn 2023-09-30T01:09:31.228 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.v6F2QIlHhn is 259KB 2023-09-30T01:09:31.322 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.v6F2QIlHhn 2023-09-30T01:09:31.340 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T01:09:31.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T01:09:31.343 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T01:09:31.407 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T01:09:31.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T01:09:31.411 DEBUG:teuthology.orchestra.run.smithi157:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T01:09:31.437 INFO:teuthology.orchestra.run.smithi115.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T01:09:31.476 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T01:09:31.482 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T01:09:31.483 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi115/crash 2023-09-30T01:09:31.484 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T01:09:31.514 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi157/crash 2023-09-30T01:09:31.515 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T01:09:31.548 INFO:tasks.ceph:Compressing logs... 2023-09-30T01:09:31.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:09:31.558 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:09:32.522 INFO:tasks.ceph:Archiving logs... 2023-09-30T01:09:32.522 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi115/log 2023-09-30T01:09:32.523 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T01:09:32.815 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi157/log 2023-09-30T01:09:32.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T01:09:32.927 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:09:32.995 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7faddf6e584c46e2960afb0f7e2afdf4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:09:32.998 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T01:09:33.016 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T01:09:33.045 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T01:09:33.045 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:09:33.057 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:09:33.137 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T01:09:33.146 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T01:09:33.146 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:09:33.149 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:09:33.164 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-09-30T01:09:33.167 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -157us[ -157us] +/- 72ms 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -209us[ -196us] +/- 53ms 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +627us[ +627us] +/- 89ms 2023-09-30T01:09:33.169 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:09:33.171 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:09:33.171 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-09-30T01:09:33.172 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 -152us[ -152us] +/- 72ms 2023-09-30T01:09:33.172 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -218us[ -174us] +/- 53ms 2023-09-30T01:09:33.172 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +631us[ +631us] +/- 89ms 2023-09-30T01:09:33.172 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:09:33.173 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T01:09:33.181 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T01:09:33.181 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T01:09:33.221 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T01:09:33.314 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T01:09:33.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:09:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:09:33.441 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:09:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:09:33.475 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T01:09:33.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T01:09:33.512 INFO:teuthology.task.internal:Duration was 1086.973780 seconds 2023-09-30T01:09:33.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T01:09:33.554 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T01:09:33.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:09:33.557 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:09:33.602 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:09:33.604 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:09:33.934 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T01:09:33.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T01:09:33.934 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:09:33.957 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T01:09:33.957 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:09:33.980 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T01:09:33.981 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:09:33.999 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:09:34.059 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T01:09:34.059 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:09:34.078 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:09:34.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-09-30T01:09:34.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-30T01:09:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-30T01:09:34.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T01:09:34.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T01:09:34.176 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T01:09:34.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:09:34.204 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:09:34.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T01:09:34.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:34.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:34.271 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2023-09-30T01:09:34.299 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2023-09-30T01:09:34.320 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:09:34.355 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:09:34.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T01:09:34.388 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T01:09:34.388 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi115 2023-09-30T01:09:34.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:09:34.439 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806/remote/smithi157 2023-09-30T01:09:34.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:09:34.478 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T01:09:34.479 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:09:34.481 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:09:34.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T01:09:34.541 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T01:09:34.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T01:09:34.574 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T01:09:34.574 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:09:34.577 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:09:34.592 INFO:teuthology.orchestra.run.smithi115.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 01:09 /home/ubuntu/cephtest 2023-09-30T01:09:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 01:09 /home/ubuntu/cephtest 2023-09-30T01:09:34.595 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T01:09:34.621 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T01:09:34.721 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T01:09:34.748 DEBUG:teuthology.nuke:shortname: smithi115 2023-09-30T01:09:34.749 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:09:34.774 DEBUG:teuthology.nuke:shortname: smithi157 2023-09-30T01:09:34.774 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:09:34.778 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:42:28.713986', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPk0GqMes8Qq3IX+gKj0dc7tzXhqKRmxhbppzXWrwEVrK/bWaCoqihwFWCgphmoKWY4NckYAsXjhGGM2ActeLA8='} 2023-09-30T01:09:34.802 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:42:28.712583', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFLTwF6gV082yTB3YMXebeTq3Z8SDVUY/zK81SDcvj45/0aCs9JXS8H7mD7/i7AtGoa7K0/sIYaPN1IHe50es0='} 2023-09-30T01:09:34.833 INFO:teuthology.orchestra.console.smithi115:Power off 2023-09-30T01:09:34.834 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:09:34.857 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2023-09-30T01:09:34.857 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:34.879 INFO:teuthology.orchestra.console.smithi157:Power off 2023-09-30T01:09:34.879 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:09:34.889 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-09-30T01:09:34.903 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-09-30T01:09:34.904 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:34.927 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T01:09:38.891 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:38.928 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:39.001 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-09-30T01:09:39.039 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T01:09:43.006 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:43.040 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:09:43.118 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2023-09-30T01:09:43.150 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-09-30T01:09:43.220 INFO:teuthology.orchestra.console.smithi115:Power off completed 2023-09-30T01:09:43.252 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-09-30T01:09:43.357 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} duration: 1086.9737799167633 failure_reason: '"1696035808.0253656 mgr.y (mgr.4103) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.y: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7faddf6e584c46e2960afb0f7e2afdf4 status: fail success: false 2023-09-30T01:09:43.357 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:09:43.443 INFO:teuthology.run:FAIL